Skip to content
GitLab
Projects
Groups
Snippets
Help
Loading...
Help
What's new
7
Help
Support
Community forum
Keyboard shortcuts
?
Submit feedback
Contribute to GitLab
Switch to GitLab Next
Sign in / Register
Toggle navigation
Open sidebar
Recommend.Games
Board Game Scraper
Commits
ac288176
Verified
Commit
ac288176
authored
Oct 27, 2020
by
Markus Shepherd
🙈
Browse files
account for item types without image fields
parent
a0bc3032
Changes
1
Hide whitespace changes
Inline
Side-by-side
Showing
1 changed file
with
11 additions
and
10 deletions
+11
-10
board_game_scraper/pipelines.py
board_game_scraper/pipelines.py
+11
-10
No files found.
board_game_scraper/pipelines.py
View file @
ac288176
...
...
@@ -219,16 +219,17 @@ class LimitImagesPipeline:
def
process_item
(
self
,
item
,
spider
):
"""Copy a limited number of image URLs to be downloaded from source to target."""
if
self
.
limit
is
None
or
self
.
limit
<
0
:
# copy through everything
item
[
self
.
target_field
]
=
list
(
arg_to_iter
(
item
.
get
(
self
.
source_field
)))
return
item
values
=
(
arg_to_iter
(
item
.
get
(
self
.
source_field
))
if
self
.
limit
is
None
or
self
.
limit
<
0
# copy through everything
else
()
if
not
self
.
limit
# limit is zero
else
islice
(
arg_to_iter
(
item
.
get
(
self
.
source_field
)),
self
.
limit
)
)
if
not
self
.
limit
:
# limit is zero
item
[
self
.
target_field
]
=
[]
return
item
try
:
item
[
self
.
target_field
]
=
list
(
values
)
except
Exception
:
pass
# actual limit
item
[
self
.
target_field
]
=
list
(
islice
(
arg_to_iter
(
item
.
get
(
self
.
source_field
)),
self
.
limit
)
)
return
item
Write
Preview
Markdown
is supported
0%
Try again
or
attach a new file
.
Attach a file
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Cancel
Please
register
or
sign in
to comment