fix cachedataset w persistent workers problem by deep copying#2121
Merged
rijobro merged 14 commits intoProject-MONAI:devfrom May 7, 2021
Merged
fix cachedataset w persistent workers problem by deep copying#2121rijobro merged 14 commits intoProject-MONAI:devfrom
rijobro merged 14 commits intoProject-MONAI:devfrom
Conversation
Signed-off-by: Richard Brown <33289025+rijobro@users.noreply.github.com>
Nic-Ma
reviewed
Apr 30, 2021
Contributor
Nic-Ma
left a comment
Nic-Ma
left a comment
There was a problem hiding this comment.
Looks good to me.
Could you please help run the CacheDataset tutorial twice with / without deepcopy to make sure it will not affect the training speed and memory usage?
https://github.com/Project-MONAI/tutorials/blob/master/acceleration/dataset_type_performance.ipynb
Thanks.
Signed-off-by: Richard Brown <33289025+rijobro@users.noreply.github.com>
…workers Signed-off-by: Richard Brown <33289025+rijobro@users.noreply.github.com>
Signed-off-by: Richard Brown <33289025+rijobro@users.noreply.github.com>
Contributor
|
Hi @rijobro , Do you have update on this PR? Thanks. |
Contributor
Author
|
Sorry, put the comment in a resolved conversation. All looks good to me, I'll just have a look at the github action error. |
Signed-off-by: Richard Brown <33289025+rijobro@users.noreply.github.com>
Contributor
Author
Contributor
|
looks like an issue with the latest pytype google/pytype#909 |
wyli
reviewed
May 7, 2021
Signed-off-by: Richard Brown <33289025+rijobro@users.noreply.github.com>
Signed-off-by: Richard Brown <33289025+rijobro@users.noreply.github.com>
Nic-Ma
approved these changes
May 7, 2021
Contributor
|
BTW, I tested this PR in our
Thanks. |
yanielc
pushed a commit
to yanielc/MONAI
that referenced
this pull request
May 10, 2021
…t-MONAI#2121) fix cachedataset w persistent workers problem by deep copying first non-deterministic transform Signed-off-by: Yaniel Cabrera <yaniel.cabrera@gmail.com>
yanielc
pushed a commit
to yanielc/MONAI
that referenced
this pull request
May 13, 2021
…t-MONAI#2121) fix cachedataset w persistent workers problem by deep copying first non-deterministic transform Signed-off-by: Yaniel Cabrera <yaniel.cabrera@gmail.com>
yanielc
pushed a commit
to yanielc/MONAI
that referenced
this pull request
May 13, 2021
…t-MONAI#2121) fix cachedataset w persistent workers problem by deep copying first non-deterministic transform Signed-off-by: Yaniel Cabrera <yaniel.cabrera@gmail.com>
wyli
pushed a commit
that referenced
this pull request
May 26, 2021
fix cachedataset w persistent workers problem by deep copying first non-deterministic transform
wyli
pushed a commit
that referenced
this pull request
May 26, 2021
fix cachedataset w persistent workers problem by deep copying first non-deterministic transform
wyli
pushed a commit
that referenced
this pull request
May 26, 2021
fix cachedataset w persistent workers problem by deep copying first non-deterministic transform
wyli
pushed a commit
that referenced
this pull request
May 26, 2021
fix cachedataset w persistent workers problem by deep copying first non-deterministic transform
wyli
pushed a commit
that referenced
this pull request
May 26, 2021
fix cachedataset w persistent workers problem by deep copying first non-deterministic transform
wyli
pushed a commit
that referenced
this pull request
May 27, 2021
fix cachedataset w persistent workers problem by deep copying first non-deterministic transform
wyli
pushed a commit
that referenced
this pull request
May 27, 2021
fix cachedataset w persistent workers problem by deep copying first non-deterministic transform
wyli
pushed a commit
that referenced
this pull request
May 27, 2021
fix cachedataset w persistent workers problem by deep copying first non-deterministic transform
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #2116.
Description
When using
CacheDatasetcombined withpersistent_workers=Truein theDataLoader, it seems that the<key>_transformsare duplicated for the first random transform onwards.So for a transform of
Compose([Spacingd, RandAffined, ToTensord]), then on the second pass through the data, we would haveimg_transformsas:Spacing, RandAffined, ToTensord, RandAffined, ToTensord. This implies a shallow copy at some point.I've put the copy in the
CacheDatasetbut I worry that this may cause unnecessary copying of the data. Having said that, I can't see where else to put it... I'm open to ideas.Status
Ready
Types of changes
./runtests.sh -f -u --net --coverage../runtests.sh --quick --unittests.make htmlcommand in thedocs/folder.