Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 19.075414014s trace: https://jaeger.dev.storj.io/trace/13f785634570d9fc FAIL (2025-12-16 03:22:41 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 16 03:12:18.507971969 (251h43m18.379055014s) [upload=220713]: Starting upload Dec 16 03:12:18.508025069 (251h43m18.379108115s) [upload=220713]: Done starting upload Dec 16 03:12:18.508197317 (251h43m18.379280353s) [upload=220713]: Uploading segments... Dec 16 03:12:18.508315022 (251h43m18.379398067s) [upload=220713]: Got next segment. Inline: false Dec 16 03:12:18.509349181 (251h43m18.380432216s) [upload=220713]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 03:12:18.509360643 (251h43m18.380443688s) [upload=220713]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 03:12:18.761879946 (251h43m18.632962991s) [seg_pos={0 0}, upload=220713]: Begin upload segment... Dec 16 03:12:18.761929029 (251h43m18.633012064s) [seg_pos={0 0}, upload=220713]: Done begin upload segment. Dec 16 03:12:18.761967552 (251h43m18.633050597s) [noise=true , address=77.90.31.10:4520 , piece_id=Q55PSEFHSAOGY7GD4DWI2G56U66BXVZHXZ52DX4LYV4B52F5ZZ6A, seg_pos={0 0}, upload=220713]: Uploading piece... Dec 16 03:12:18.762560491 (251h43m18.633643536s) [noise=true , address=45.91.116.6:4974 , piece_id=GQDHG4LFMNUHAKFA3GA3WANCW5OGHOWCYM3IZCUGTWI54LGFWDDA, seg_pos={0 0}, upload=220713]: Uploading piece... Dec 16 03:12:18.762660291 (251h43m18.633743336s) [noise=true , address=145.79.227.25:4617 , piece_id=TCLE2SK75NCWPLGUPYLJI3M5WD75Z2APQUGUXYJFFYAH5MU42T6A, seg_pos={0 0}, upload=220713]: Uploading piece... Dec 16 03:12:18.763036347 (251h43m18.634119382s) [noise=true , address=145.79.227.36:4518 , piece_id=I5C4LFPXUEB2JBUTWHXQV7VRJCCNN2RP7Z3CQ7RQENNMUBNNMHYQ, seg_pos={0 0}, upload=220713]: Uploading piece... Dec 16 03:12:18.763318324 (251h43m18.634401359s) [noise=true , address=145.79.227.14:4840 , piece_id=3ABV6MR5KUUYYF7I2TU5MEXF5ULEY6IU6YIF72DCGR5MBALRWF3A, seg_pos={0 0}, upload=220713]: Uploading piece... Dec 16 03:12:18.763607314 (251h43m18.634690349s) [noise=true , address=145.79.227.50:4373 , piece_id=YMG7Q5VQQKAYHO2HAEPOBRRDARQ25HGALNK3YOMIXUXYQONNKXSA, seg_pos={0 0}, upload=220713]: Uploading piece... Dec 16 03:12:18.763864645 (251h43m18.63494769s) [noise=true , address=145.79.227.51:4409 , piece_id=3IEX2COS67O2QHKMMPC2NZJ3PGDRDG3O5WOKIY5IA4HKMZFDNF4Q, seg_pos={0 0}, upload=220713]: Uploading piece... Dec 16 03:12:18.763888079 (251h43m18.634971154s) [noise=true , address=145.79.227.42:4123 , piece_id=NMJHDWWFESF3GM5TMEZFQGFD7YY5QHIIYQVHF3JIINSJHSEQDFZQ, seg_pos={0 0}, upload=220713]: Uploading piece... Dec 16 03:12:18.763947542 (251h43m18.635030607s) [noise=true , address=145.79.227.32:4834 , piece_id=RRMTYRQ37P6YWUV6BPVUC2BFLCRPWN6Q7VEVX6BDA54IKWTFH7SQ, seg_pos={0 0}, upload=220713]: Uploading piece... Dec 16 03:12:18.764263093 (251h43m18.635346138s) [noise=true , address=145.79.227.37:4565 , piece_id=4VFLWAOYZ7OOHBEDWSBLAWDZODVQEU3MFCQKSNGUE6HLC3MDPVCQ, seg_pos={0 0}, upload=220713]: Uploading piece... Dec 16 03:12:18.764390716 (251h43m18.635473761s) [noise=true , address=145.79.227.44:4194 , piece_id=RJPVGEHG2XIR5JAPDFN6FR5ZH7ICFU2ZQIQKURWSLTBQT4UV7POA, seg_pos={0 0}, upload=220713]: Uploading piece... Dec 16 03:12:18.764941455 (251h43m18.6360245s) [noise=true , address=145.79.227.33:4870 , piece_id=MRUDYCE63CU2SHIFP64BNHFRLYAO2A3KXN6GIZGG7QJGX6S7UXDA, seg_pos={0 0}, upload=220713]: Uploading piece... Dec 16 03:12:18.765504287 (251h43m18.636587322s) [noise=true , address=45.91.116.5:4298 , piece_id=GVZCYRIPTXYE6WOPQPRYKJSGV23I3GZ22SOCOJSXFUBLNDVQN2MA, seg_pos={0 0}, upload=220713]: Uploading piece... Dec 16 03:12:18.766127143 (251h43m18.637210188s) [noise=true , address=145.79.227.29:4942 , piece_id=SON2DHJWJHPLZ7IJG6HFLK6D6C5O6JL2DMLLPOB5GHSPET544HAA, seg_pos={0 0}, upload=220713]: Uploading piece... Dec 16 03:12:18.766432894 (251h43m18.637515929s) [noise=true , address=77.90.31.16:4735 , piece_id=ATZ3AJVXFRAXDM4DR6MKDEHTRHM6AG4RB4YLKDHDLDMO5XFF3MPA, seg_pos={0 0}, upload=220713]: Uploading piece... Dec 16 03:12:18.766736152 (251h43m18.637819187s) [noise=true , address=145.79.227.52:4447 , piece_id=OYGRYLI43OEMB7MBUTGLXG76U4WUUR3EMNCUZQ5C67DVHLWBVHJA, seg_pos={0 0}, upload=220713]: Uploading piece... Dec 16 03:12:20.470095008 (251h43m20.341178064s) [upload=220713]: Next returned nil segment Dec 16 03:12:20.470146787 (251h43m20.341229832s) [upload=220713]: Waiting for error group managing segments... Dec 16 03:12:20.753665772 (251h43m20.624748807s) [noise=true , address=145.79.227.42:4123 , piece_id=NMJHDWWFESF3GM5TMEZFQGFD7YY5QHIIYQVHF3JIINSJHSEQDFZQ, seg_pos={0 0}, upload=220713]: Done uploading piece. err: <nil> Dec 16 03:12:20.757729490 (251h43m20.628812535s) [noise=true , address=145.79.227.29:4942 , piece_id=SON2DHJWJHPLZ7IJG6HFLK6D6C5O6JL2DMLLPOB5GHSPET544HAA, seg_pos={0 0}, upload=220713]: Done uploading piece. err: <nil> Dec 16 03:12:20.760602501 (251h43m20.631685546s) [noise=true , address=145.79.227.36:4518 , piece_id=I5C4LFPXUEB2JBUTWHXQV7VRJCCNN2RP7Z3CQ7RQENNMUBNNMHYQ, seg_pos={0 0}, upload=220713]: Done uploading piece. err: <nil> Dec 16 03:12:20.764326061 (251h43m20.635409106s) [noise=true , address=145.79.227.25:4617 , piece_id=TCLE2SK75NCWPLGUPYLJI3M5WD75Z2APQUGUXYJFFYAH5MU42T6A, seg_pos={0 0}, upload=220713]: Done uploading piece. err: <nil> Dec 16 03:12:20.766406653 (251h43m20.637489698s) [noise=true , address=145.79.227.37:4565 , piece_id=4VFLWAOYZ7OOHBEDWSBLAWDZODVQEU3MFCQKSNGUE6HLC3MDPVCQ, seg_pos={0 0}, upload=220713]: Done uploading piece. err: <nil> Dec 16 03:12:20.770108361 (251h43m20.641191406s) [noise=true , address=145.79.227.33:4870 , piece_id=MRUDYCE63CU2SHIFP64BNHFRLYAO2A3KXN6GIZGG7QJGX6S7UXDA, seg_pos={0 0}, upload=220713]: Done uploading piece. err: <nil> Dec 16 03:12:20.773437491 (251h43m20.644520546s) [noise=true , address=145.79.227.50:4373 , piece_id=YMG7Q5VQQKAYHO2HAEPOBRRDARQ25HGALNK3YOMIXUXYQONNKXSA, seg_pos={0 0}, upload=220713]: Done uploading piece. err: <nil> Dec 16 03:12:20.775082453 (251h43m20.646165488s) [noise=true , address=145.79.227.52:4447 , piece_id=OYGRYLI43OEMB7MBUTGLXG76U4WUUR3EMNCUZQ5C67DVHLWBVHJA, seg_pos={0 0}, upload=220713]: Done uploading piece. err: <nil> Dec 16 03:12:20.833660625 (251h43m20.70474366s) [noise=true , address=145.79.227.51:4409 , piece_id=3IEX2COS67O2QHKMMPC2NZJ3PGDRDG3O5WOKIY5IA4HKMZFDNF4Q, seg_pos={0 0}, upload=220713]: Done uploading piece. err: <nil> Dec 16 03:12:20.844657545 (251h43m20.71574058s) [noise=true , address=145.79.227.14:4840 , piece_id=3ABV6MR5KUUYYF7I2TU5MEXF5ULEY6IU6YIF72DCGR5MBALRWF3A, seg_pos={0 0}, upload=220713]: Done uploading piece. err: <nil> Dec 16 03:12:20.859069218 (251h43m20.730152273s) [noise=true , address=145.79.227.32:4834 , piece_id=RRMTYRQ37P6YWUV6BPVUC2BFLCRPWN6Q7VEVX6BDA54IKWTFH7SQ, seg_pos={0 0}, upload=220713]: Done uploading piece. err: <nil> Dec 16 03:12:21.007293480 (251h43m20.878376515s) [noise=true , address=77.90.31.16:4735 , piece_id=ATZ3AJVXFRAXDM4DR6MKDEHTRHM6AG4RB4YLKDHDLDMO5XFF3MPA, seg_pos={0 0}, upload=220713]: Done uploading piece. err: <nil> Dec 16 03:12:21.014381907 (251h43m20.885464952s) [noise=true , address=45.91.116.5:4298 , piece_id=GVZCYRIPTXYE6WOPQPRYKJSGV23I3GZ22SOCOJSXFUBLNDVQN2MA, seg_pos={0 0}, upload=220713]: Done uploading piece. err: <nil> Dec 16 03:12:21.014403138 (251h43m20.885486173s) [seg_pos={0 0}, upload=220713]: Segment reached optimal threshold of 13 pieces. Dec 16 03:12:21.014711706 (251h43m20.885794741s) [noise=true , address=145.79.227.44:4194 , piece_id=RJPVGEHG2XIR5JAPDFN6FR5ZH7ICFU2ZQIQKURWSLTBQT4UV7POA, seg_pos={0 0}, upload=220713]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh): protocol: expected piece hash; context canceled; EOF Dec 16 03:12:21.014721354 (251h43m20.885804389s) [noise=true , address=145.79.227.44:4194 , piece_id=RJPVGEHG2XIR5JAPDFN6FR5ZH7ICFU2ZQIQKURWSLTBQT4UV7POA, seg_pos={0 0}, upload=220713]: Long tail cancellation due to optimal threshold Dec 16 03:12:21.014764125 (251h43m20.88584716s) [noise=true , address=45.91.116.6:4974 , piece_id=GQDHG4LFMNUHAKFA3GA3WANCW5OGHOWCYM3IZCUGTWI54LGFWDDA, seg_pos={0 0}, upload=220713]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: expected piece hash; context canceled; EOF Dec 16 03:12:21.014782741 (251h43m20.885865786s) [noise=true , address=45.91.116.6:4974 , piece_id=GQDHG4LFMNUHAKFA3GA3WANCW5OGHOWCYM3IZCUGTWI54LGFWDDA, seg_pos={0 0}, upload=220713]: Long tail cancellation due to optimal threshold Dec 16 03:12:21.014804422 (251h43m20.885887467s) [noise=true , address=77.90.31.10:4520 , piece_id=Q55PSEFHSAOGY7GD4DWI2G56U66BXVZHXZ52DX4LYV4B52F5ZZ6A, seg_pos={0 0}, upload=220713]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp): protocol: expected piece hash; context canceled; EOF Dec 16 03:12:21.014809482 (251h43m20.885892527s) [noise=true , address=77.90.31.10:4520 , piece_id=Q55PSEFHSAOGY7GD4DWI2G56U66BXVZHXZ52DX4LYV4B52F5ZZ6A, seg_pos={0 0}, upload=220713]: Long tail cancellation due to optimal threshold Dec 16 03:12:21.014826394 (251h43m20.885909429s) [seg_pos={0 0}, upload=220713]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 03:12:21.015234772 (251h43m20.886317807s) [upload=220713]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 03:12:21.015241204 (251h43m20.886324249s) [upload=220713]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 03:12:21.319403538 (251h43m21.190486583s) [upload=220713]: Done uploading segments... Dec 16 03:12:21.319427222 (251h43m21.190510257s) [upload=220713]: Upload finished. err: <nil> Duration: 19.075439301s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 6.027136109s trace: https://jaeger.dev.storj.io/trace/2e0c5260bf6208b3 FAIL (2025-12-16 03:22:20 UTC) orbiter warning: context deadline exceeded Duration: 6.027170546s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 6.147060746s trace: https://jaeger.dev.storj.io/trace/2e0c5260bf6208b3 FAIL (2025-12-16 03:21:47 UTC) orbiter warning: context deadline exceeded Duration: 6.147108277s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 5.987470191s trace: https://jaeger.dev.storj.io/trace/13f785634570d9fc FAIL (2025-12-16 03:21:45 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 16 03:12:18.507971969 (251h43m18.379055014s) [upload=220713]: Starting upload Dec 16 03:12:18.508025069 (251h43m18.379108115s) [upload=220713]: Done starting upload Dec 16 03:12:18.508197317 (251h43m18.379280353s) [upload=220713]: Uploading segments... Dec 16 03:12:18.508315022 (251h43m18.379398067s) [upload=220713]: Got next segment. Inline: false Dec 16 03:12:18.509349181 (251h43m18.380432216s) [upload=220713]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 03:12:18.509360643 (251h43m18.380443688s) [upload=220713]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 03:12:18.761879946 (251h43m18.632962991s) [seg_pos={0 0}, upload=220713]: Begin upload segment... Dec 16 03:12:18.761929029 (251h43m18.633012064s) [seg_pos={0 0}, upload=220713]: Done begin upload segment. Dec 16 03:12:18.761967552 (251h43m18.633050597s) [noise=true , address=77.90.31.10:4520 , piece_id=Q55PSEFHSAOGY7GD4DWI2G56U66BXVZHXZ52DX4LYV4B52F5ZZ6A, seg_pos={0 0}, upload=220713]: Uploading piece... Dec 16 03:12:18.762560491 (251h43m18.633643536s) [noise=true , address=45.91.116.6:4974 , piece_id=GQDHG4LFMNUHAKFA3GA3WANCW5OGHOWCYM3IZCUGTWI54LGFWDDA, seg_pos={0 0}, upload=220713]: Uploading piece... Dec 16 03:12:18.762660291 (251h43m18.633743336s) [noise=true , address=145.79.227.25:4617 , piece_id=TCLE2SK75NCWPLGUPYLJI3M5WD75Z2APQUGUXYJFFYAH5MU42T6A, seg_pos={0 0}, upload=220713]: Uploading piece... Dec 16 03:12:18.763036347 (251h43m18.634119382s) [noise=true , address=145.79.227.36:4518 , piece_id=I5C4LFPXUEB2JBUTWHXQV7VRJCCNN2RP7Z3CQ7RQENNMUBNNMHYQ, seg_pos={0 0}, upload=220713]: Uploading piece... Dec 16 03:12:18.763318324 (251h43m18.634401359s) [noise=true , address=145.79.227.14:4840 , piece_id=3ABV6MR5KUUYYF7I2TU5MEXF5ULEY6IU6YIF72DCGR5MBALRWF3A, seg_pos={0 0}, upload=220713]: Uploading piece... Dec 16 03:12:18.763607314 (251h43m18.634690349s) [noise=true , address=145.79.227.50:4373 , piece_id=YMG7Q5VQQKAYHO2HAEPOBRRDARQ25HGALNK3YOMIXUXYQONNKXSA, seg_pos={0 0}, upload=220713]: Uploading piece... Dec 16 03:12:18.763864645 (251h43m18.63494769s) [noise=true , address=145.79.227.51:4409 , piece_id=3IEX2COS67O2QHKMMPC2NZJ3PGDRDG3O5WOKIY5IA4HKMZFDNF4Q, seg_pos={0 0}, upload=220713]: Uploading piece... Dec 16 03:12:18.763888079 (251h43m18.634971154s) [noise=true , address=145.79.227.42:4123 , piece_id=NMJHDWWFESF3GM5TMEZFQGFD7YY5QHIIYQVHF3JIINSJHSEQDFZQ, seg_pos={0 0}, upload=220713]: Uploading piece... Dec 16 03:12:18.763947542 (251h43m18.635030607s) [noise=true , address=145.79.227.32:4834 , piece_id=RRMTYRQ37P6YWUV6BPVUC2BFLCRPWN6Q7VEVX6BDA54IKWTFH7SQ, seg_pos={0 0}, upload=220713]: Uploading piece... Dec 16 03:12:18.764263093 (251h43m18.635346138s) [noise=true , address=145.79.227.37:4565 , piece_id=4VFLWAOYZ7OOHBEDWSBLAWDZODVQEU3MFCQKSNGUE6HLC3MDPVCQ, seg_pos={0 0}, upload=220713]: Uploading piece... Dec 16 03:12:18.764390716 (251h43m18.635473761s) [noise=true , address=145.79.227.44:4194 , piece_id=RJPVGEHG2XIR5JAPDFN6FR5ZH7ICFU2ZQIQKURWSLTBQT4UV7POA, seg_pos={0 0}, upload=220713]: Uploading piece... Dec 16 03:12:18.764941455 (251h43m18.6360245s) [noise=true , address=145.79.227.33:4870 , piece_id=MRUDYCE63CU2SHIFP64BNHFRLYAO2A3KXN6GIZGG7QJGX6S7UXDA, seg_pos={0 0}, upload=220713]: Uploading piece... Dec 16 03:12:18.765504287 (251h43m18.636587322s) [noise=true , address=45.91.116.5:4298 , piece_id=GVZCYRIPTXYE6WOPQPRYKJSGV23I3GZ22SOCOJSXFUBLNDVQN2MA, seg_pos={0 0}, upload=220713]: Uploading piece... Dec 16 03:12:18.766127143 (251h43m18.637210188s) [noise=true , address=145.79.227.29:4942 , piece_id=SON2DHJWJHPLZ7IJG6HFLK6D6C5O6JL2DMLLPOB5GHSPET544HAA, seg_pos={0 0}, upload=220713]: Uploading piece... Dec 16 03:12:18.766432894 (251h43m18.637515929s) [noise=true , address=77.90.31.16:4735 , piece_id=ATZ3AJVXFRAXDM4DR6MKDEHTRHM6AG4RB4YLKDHDLDMO5XFF3MPA, seg_pos={0 0}, upload=220713]: Uploading piece... Dec 16 03:12:18.766736152 (251h43m18.637819187s) [noise=true , address=145.79.227.52:4447 , piece_id=OYGRYLI43OEMB7MBUTGLXG76U4WUUR3EMNCUZQ5C67DVHLWBVHJA, seg_pos={0 0}, upload=220713]: Uploading piece... Dec 16 03:12:20.470095008 (251h43m20.341178064s) [upload=220713]: Next returned nil segment Dec 16 03:12:20.470146787 (251h43m20.341229832s) [upload=220713]: Waiting for error group managing segments... Dec 16 03:12:20.753665772 (251h43m20.624748807s) [noise=true , address=145.79.227.42:4123 , piece_id=NMJHDWWFESF3GM5TMEZFQGFD7YY5QHIIYQVHF3JIINSJHSEQDFZQ, seg_pos={0 0}, upload=220713]: Done uploading piece. err: <nil> Dec 16 03:12:20.757729490 (251h43m20.628812535s) [noise=true , address=145.79.227.29:4942 , piece_id=SON2DHJWJHPLZ7IJG6HFLK6D6C5O6JL2DMLLPOB5GHSPET544HAA, seg_pos={0 0}, upload=220713]: Done uploading piece. err: <nil> Dec 16 03:12:20.760602501 (251h43m20.631685546s) [noise=true , address=145.79.227.36:4518 , piece_id=I5C4LFPXUEB2JBUTWHXQV7VRJCCNN2RP7Z3CQ7RQENNMUBNNMHYQ, seg_pos={0 0}, upload=220713]: Done uploading piece. err: <nil> Dec 16 03:12:20.764326061 (251h43m20.635409106s) [noise=true , address=145.79.227.25:4617 , piece_id=TCLE2SK75NCWPLGUPYLJI3M5WD75Z2APQUGUXYJFFYAH5MU42T6A, seg_pos={0 0}, upload=220713]: Done uploading piece. err: <nil> Dec 16 03:12:20.766406653 (251h43m20.637489698s) [noise=true , address=145.79.227.37:4565 , piece_id=4VFLWAOYZ7OOHBEDWSBLAWDZODVQEU3MFCQKSNGUE6HLC3MDPVCQ, seg_pos={0 0}, upload=220713]: Done uploading piece. err: <nil> Dec 16 03:12:20.770108361 (251h43m20.641191406s) [noise=true , address=145.79.227.33:4870 , piece_id=MRUDYCE63CU2SHIFP64BNHFRLYAO2A3KXN6GIZGG7QJGX6S7UXDA, seg_pos={0 0}, upload=220713]: Done uploading piece. err: <nil> Dec 16 03:12:20.773437491 (251h43m20.644520546s) [noise=true , address=145.79.227.50:4373 , piece_id=YMG7Q5VQQKAYHO2HAEPOBRRDARQ25HGALNK3YOMIXUXYQONNKXSA, seg_pos={0 0}, upload=220713]: Done uploading piece. err: <nil> Dec 16 03:12:20.775082453 (251h43m20.646165488s) [noise=true , address=145.79.227.52:4447 , piece_id=OYGRYLI43OEMB7MBUTGLXG76U4WUUR3EMNCUZQ5C67DVHLWBVHJA, seg_pos={0 0}, upload=220713]: Done uploading piece. err: <nil> Dec 16 03:12:20.833660625 (251h43m20.70474366s) [noise=true , address=145.79.227.51:4409 , piece_id=3IEX2COS67O2QHKMMPC2NZJ3PGDRDG3O5WOKIY5IA4HKMZFDNF4Q, seg_pos={0 0}, upload=220713]: Done uploading piece. err: <nil> Dec 16 03:12:20.844657545 (251h43m20.71574058s) [noise=true , address=145.79.227.14:4840 , piece_id=3ABV6MR5KUUYYF7I2TU5MEXF5ULEY6IU6YIF72DCGR5MBALRWF3A, seg_pos={0 0}, upload=220713]: Done uploading piece. err: <nil> Dec 16 03:12:20.859069218 (251h43m20.730152273s) [noise=true , address=145.79.227.32:4834 , piece_id=RRMTYRQ37P6YWUV6BPVUC2BFLCRPWN6Q7VEVX6BDA54IKWTFH7SQ, seg_pos={0 0}, upload=220713]: Done uploading piece. err: <nil> Dec 16 03:12:21.007293480 (251h43m20.878376515s) [noise=true , address=77.90.31.16:4735 , piece_id=ATZ3AJVXFRAXDM4DR6MKDEHTRHM6AG4RB4YLKDHDLDMO5XFF3MPA, seg_pos={0 0}, upload=220713]: Done uploading piece. err: <nil> Dec 16 03:12:21.014381907 (251h43m20.885464952s) [noise=true , address=45.91.116.5:4298 , piece_id=GVZCYRIPTXYE6WOPQPRYKJSGV23I3GZ22SOCOJSXFUBLNDVQN2MA, seg_pos={0 0}, upload=220713]: Done uploading piece. err: <nil> Dec 16 03:12:21.014403138 (251h43m20.885486173s) [seg_pos={0 0}, upload=220713]: Segment reached optimal threshold of 13 pieces. Dec 16 03:12:21.014711706 (251h43m20.885794741s) [noise=true , address=145.79.227.44:4194 , piece_id=RJPVGEHG2XIR5JAPDFN6FR5ZH7ICFU2ZQIQKURWSLTBQT4UV7POA, seg_pos={0 0}, upload=220713]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh): protocol: expected piece hash; context canceled; EOF Dec 16 03:12:21.014721354 (251h43m20.885804389s) [noise=true , address=145.79.227.44:4194 , piece_id=RJPVGEHG2XIR5JAPDFN6FR5ZH7ICFU2ZQIQKURWSLTBQT4UV7POA, seg_pos={0 0}, upload=220713]: Long tail cancellation due to optimal threshold Dec 16 03:12:21.014764125 (251h43m20.88584716s) [noise=true , address=45.91.116.6:4974 , piece_id=GQDHG4LFMNUHAKFA3GA3WANCW5OGHOWCYM3IZCUGTWI54LGFWDDA, seg_pos={0 0}, upload=220713]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: expected piece hash; context canceled; EOF Dec 16 03:12:21.014782741 (251h43m20.885865786s) [noise=true , address=45.91.116.6:4974 , piece_id=GQDHG4LFMNUHAKFA3GA3WANCW5OGHOWCYM3IZCUGTWI54LGFWDDA, seg_pos={0 0}, upload=220713]: Long tail cancellation due to optimal threshold Dec 16 03:12:21.014804422 (251h43m20.885887467s) [noise=true , address=77.90.31.10:4520 , piece_id=Q55PSEFHSAOGY7GD4DWI2G56U66BXVZHXZ52DX4LYV4B52F5ZZ6A, seg_pos={0 0}, upload=220713]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp): protocol: expected piece hash; context canceled; EOF Dec 16 03:12:21.014809482 (251h43m20.885892527s) [noise=true , address=77.90.31.10:4520 , piece_id=Q55PSEFHSAOGY7GD4DWI2G56U66BXVZHXZ52DX4LYV4B52F5ZZ6A, seg_pos={0 0}, upload=220713]: Long tail cancellation due to optimal threshold Dec 16 03:12:21.014826394 (251h43m20.885909429s) [seg_pos={0 0}, upload=220713]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 03:12:21.015234772 (251h43m20.886317807s) [upload=220713]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 03:12:21.015241204 (251h43m20.886324249s) [upload=220713]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 03:12:21.319403538 (251h43m21.190486583s) [upload=220713]: Done uploading segments... Dec 16 03:12:21.319427222 (251h43m21.190510257s) [upload=220713]: Upload finished. err: <nil> Duration: 5.987493646s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 5.87151916s trace: https://jaeger.dev.storj.io/trace/13f785634570d9fc FAIL (2025-12-16 03:21:10 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 16 03:12:18.507971969 (251h43m18.379055014s) [upload=220713]: Starting upload Dec 16 03:12:18.508025069 (251h43m18.379108115s) [upload=220713]: Done starting upload Dec 16 03:12:18.508197317 (251h43m18.379280353s) [upload=220713]: Uploading segments... Dec 16 03:12:18.508315022 (251h43m18.379398067s) [upload=220713]: Got next segment. Inline: false Dec 16 03:12:18.509349181 (251h43m18.380432216s) [upload=220713]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 03:12:18.509360643 (251h43m18.380443688s) [upload=220713]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 03:12:18.761879946 (251h43m18.632962991s) [seg_pos={0 0}, upload=220713]: Begin upload segment... Dec 16 03:12:18.761929029 (251h43m18.633012064s) [seg_pos={0 0}, upload=220713]: Done begin upload segment. Dec 16 03:12:18.761967552 (251h43m18.633050597s) [noise=true , address=77.90.31.10:4520 , piece_id=Q55PSEFHSAOGY7GD4DWI2G56U66BXVZHXZ52DX4LYV4B52F5ZZ6A, seg_pos={0 0}, upload=220713]: Uploading piece... Dec 16 03:12:18.762560491 (251h43m18.633643536s) [noise=true , address=45.91.116.6:4974 , piece_id=GQDHG4LFMNUHAKFA3GA3WANCW5OGHOWCYM3IZCUGTWI54LGFWDDA, seg_pos={0 0}, upload=220713]: Uploading piece... Dec 16 03:12:18.762660291 (251h43m18.633743336s) [noise=true , address=145.79.227.25:4617 , piece_id=TCLE2SK75NCWPLGUPYLJI3M5WD75Z2APQUGUXYJFFYAH5MU42T6A, seg_pos={0 0}, upload=220713]: Uploading piece... Dec 16 03:12:18.763036347 (251h43m18.634119382s) [noise=true , address=145.79.227.36:4518 , piece_id=I5C4LFPXUEB2JBUTWHXQV7VRJCCNN2RP7Z3CQ7RQENNMUBNNMHYQ, seg_pos={0 0}, upload=220713]: Uploading piece... Dec 16 03:12:18.763318324 (251h43m18.634401359s) [noise=true , address=145.79.227.14:4840 , piece_id=3ABV6MR5KUUYYF7I2TU5MEXF5ULEY6IU6YIF72DCGR5MBALRWF3A, seg_pos={0 0}, upload=220713]: Uploading piece... Dec 16 03:12:18.763607314 (251h43m18.634690349s) [noise=true , address=145.79.227.50:4373 , piece_id=YMG7Q5VQQKAYHO2HAEPOBRRDARQ25HGALNK3YOMIXUXYQONNKXSA, seg_pos={0 0}, upload=220713]: Uploading piece... Dec 16 03:12:18.763864645 (251h43m18.63494769s) [noise=true , address=145.79.227.51:4409 , piece_id=3IEX2COS67O2QHKMMPC2NZJ3PGDRDG3O5WOKIY5IA4HKMZFDNF4Q, seg_pos={0 0}, upload=220713]: Uploading piece... Dec 16 03:12:18.763888079 (251h43m18.634971154s) [noise=true , address=145.79.227.42:4123 , piece_id=NMJHDWWFESF3GM5TMEZFQGFD7YY5QHIIYQVHF3JIINSJHSEQDFZQ, seg_pos={0 0}, upload=220713]: Uploading piece... Dec 16 03:12:18.763947542 (251h43m18.635030607s) [noise=true , address=145.79.227.32:4834 , piece_id=RRMTYRQ37P6YWUV6BPVUC2BFLCRPWN6Q7VEVX6BDA54IKWTFH7SQ, seg_pos={0 0}, upload=220713]: Uploading piece... Dec 16 03:12:18.764263093 (251h43m18.635346138s) [noise=true , address=145.79.227.37:4565 , piece_id=4VFLWAOYZ7OOHBEDWSBLAWDZODVQEU3MFCQKSNGUE6HLC3MDPVCQ, seg_pos={0 0}, upload=220713]: Uploading piece... Dec 16 03:12:18.764390716 (251h43m18.635473761s) [noise=true , address=145.79.227.44:4194 , piece_id=RJPVGEHG2XIR5JAPDFN6FR5ZH7ICFU2ZQIQKURWSLTBQT4UV7POA, seg_pos={0 0}, upload=220713]: Uploading piece... Dec 16 03:12:18.764941455 (251h43m18.6360245s) [noise=true , address=145.79.227.33:4870 , piece_id=MRUDYCE63CU2SHIFP64BNHFRLYAO2A3KXN6GIZGG7QJGX6S7UXDA, seg_pos={0 0}, upload=220713]: Uploading piece... Dec 16 03:12:18.765504287 (251h43m18.636587322s) [noise=true , address=45.91.116.5:4298 , piece_id=GVZCYRIPTXYE6WOPQPRYKJSGV23I3GZ22SOCOJSXFUBLNDVQN2MA, seg_pos={0 0}, upload=220713]: Uploading piece... Dec 16 03:12:18.766127143 (251h43m18.637210188s) [noise=true , address=145.79.227.29:4942 , piece_id=SON2DHJWJHPLZ7IJG6HFLK6D6C5O6JL2DMLLPOB5GHSPET544HAA, seg_pos={0 0}, upload=220713]: Uploading piece... Dec 16 03:12:18.766432894 (251h43m18.637515929s) [noise=true , address=77.90.31.16:4735 , piece_id=ATZ3AJVXFRAXDM4DR6MKDEHTRHM6AG4RB4YLKDHDLDMO5XFF3MPA, seg_pos={0 0}, upload=220713]: Uploading piece... Dec 16 03:12:18.766736152 (251h43m18.637819187s) [noise=true , address=145.79.227.52:4447 , piece_id=OYGRYLI43OEMB7MBUTGLXG76U4WUUR3EMNCUZQ5C67DVHLWBVHJA, seg_pos={0 0}, upload=220713]: Uploading piece... Dec 16 03:12:20.470095008 (251h43m20.341178064s) [upload=220713]: Next returned nil segment Dec 16 03:12:20.470146787 (251h43m20.341229832s) [upload=220713]: Waiting for error group managing segments... Dec 16 03:12:20.753665772 (251h43m20.624748807s) [noise=true , address=145.79.227.42:4123 , piece_id=NMJHDWWFESF3GM5TMEZFQGFD7YY5QHIIYQVHF3JIINSJHSEQDFZQ, seg_pos={0 0}, upload=220713]: Done uploading piece. err: <nil> Dec 16 03:12:20.757729490 (251h43m20.628812535s) [noise=true , address=145.79.227.29:4942 , piece_id=SON2DHJWJHPLZ7IJG6HFLK6D6C5O6JL2DMLLPOB5GHSPET544HAA, seg_pos={0 0}, upload=220713]: Done uploading piece. err: <nil> Dec 16 03:12:20.760602501 (251h43m20.631685546s) [noise=true , address=145.79.227.36:4518 , piece_id=I5C4LFPXUEB2JBUTWHXQV7VRJCCNN2RP7Z3CQ7RQENNMUBNNMHYQ, seg_pos={0 0}, upload=220713]: Done uploading piece. err: <nil> Dec 16 03:12:20.764326061 (251h43m20.635409106s) [noise=true , address=145.79.227.25:4617 , piece_id=TCLE2SK75NCWPLGUPYLJI3M5WD75Z2APQUGUXYJFFYAH5MU42T6A, seg_pos={0 0}, upload=220713]: Done uploading piece. err: <nil> Dec 16 03:12:20.766406653 (251h43m20.637489698s) [noise=true , address=145.79.227.37:4565 , piece_id=4VFLWAOYZ7OOHBEDWSBLAWDZODVQEU3MFCQKSNGUE6HLC3MDPVCQ, seg_pos={0 0}, upload=220713]: Done uploading piece. err: <nil> Dec 16 03:12:20.770108361 (251h43m20.641191406s) [noise=true , address=145.79.227.33:4870 , piece_id=MRUDYCE63CU2SHIFP64BNHFRLYAO2A3KXN6GIZGG7QJGX6S7UXDA, seg_pos={0 0}, upload=220713]: Done uploading piece. err: <nil> Dec 16 03:12:20.773437491 (251h43m20.644520546s) [noise=true , address=145.79.227.50:4373 , piece_id=YMG7Q5VQQKAYHO2HAEPOBRRDARQ25HGALNK3YOMIXUXYQONNKXSA, seg_pos={0 0}, upload=220713]: Done uploading piece. err: <nil> Dec 16 03:12:20.775082453 (251h43m20.646165488s) [noise=true , address=145.79.227.52:4447 , piece_id=OYGRYLI43OEMB7MBUTGLXG76U4WUUR3EMNCUZQ5C67DVHLWBVHJA, seg_pos={0 0}, upload=220713]: Done uploading piece. err: <nil> Dec 16 03:12:20.833660625 (251h43m20.70474366s) [noise=true , address=145.79.227.51:4409 , piece_id=3IEX2COS67O2QHKMMPC2NZJ3PGDRDG3O5WOKIY5IA4HKMZFDNF4Q, seg_pos={0 0}, upload=220713]: Done uploading piece. err: <nil> Dec 16 03:12:20.844657545 (251h43m20.71574058s) [noise=true , address=145.79.227.14:4840 , piece_id=3ABV6MR5KUUYYF7I2TU5MEXF5ULEY6IU6YIF72DCGR5MBALRWF3A, seg_pos={0 0}, upload=220713]: Done uploading piece. err: <nil> Dec 16 03:12:20.859069218 (251h43m20.730152273s) [noise=true , address=145.79.227.32:4834 , piece_id=RRMTYRQ37P6YWUV6BPVUC2BFLCRPWN6Q7VEVX6BDA54IKWTFH7SQ, seg_pos={0 0}, upload=220713]: Done uploading piece. err: <nil> Dec 16 03:12:21.007293480 (251h43m20.878376515s) [noise=true , address=77.90.31.16:4735 , piece_id=ATZ3AJVXFRAXDM4DR6MKDEHTRHM6AG4RB4YLKDHDLDMO5XFF3MPA, seg_pos={0 0}, upload=220713]: Done uploading piece. err: <nil> Dec 16 03:12:21.014381907 (251h43m20.885464952s) [noise=true , address=45.91.116.5:4298 , piece_id=GVZCYRIPTXYE6WOPQPRYKJSGV23I3GZ22SOCOJSXFUBLNDVQN2MA, seg_pos={0 0}, upload=220713]: Done uploading piece. err: <nil> Dec 16 03:12:21.014403138 (251h43m20.885486173s) [seg_pos={0 0}, upload=220713]: Segment reached optimal threshold of 13 pieces. Dec 16 03:12:21.014711706 (251h43m20.885794741s) [noise=true , address=145.79.227.44:4194 , piece_id=RJPVGEHG2XIR5JAPDFN6FR5ZH7ICFU2ZQIQKURWSLTBQT4UV7POA, seg_pos={0 0}, upload=220713]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh): protocol: expected piece hash; context canceled; EOF Dec 16 03:12:21.014721354 (251h43m20.885804389s) [noise=true , address=145.79.227.44:4194 , piece_id=RJPVGEHG2XIR5JAPDFN6FR5ZH7ICFU2ZQIQKURWSLTBQT4UV7POA, seg_pos={0 0}, upload=220713]: Long tail cancellation due to optimal threshold Dec 16 03:12:21.014764125 (251h43m20.88584716s) [noise=true , address=45.91.116.6:4974 , piece_id=GQDHG4LFMNUHAKFA3GA3WANCW5OGHOWCYM3IZCUGTWI54LGFWDDA, seg_pos={0 0}, upload=220713]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: expected piece hash; context canceled; EOF Dec 16 03:12:21.014782741 (251h43m20.885865786s) [noise=true , address=45.91.116.6:4974 , piece_id=GQDHG4LFMNUHAKFA3GA3WANCW5OGHOWCYM3IZCUGTWI54LGFWDDA, seg_pos={0 0}, upload=220713]: Long tail cancellation due to optimal threshold Dec 16 03:12:21.014804422 (251h43m20.885887467s) [noise=true , address=77.90.31.10:4520 , piece_id=Q55PSEFHSAOGY7GD4DWI2G56U66BXVZHXZ52DX4LYV4B52F5ZZ6A, seg_pos={0 0}, upload=220713]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp): protocol: expected piece hash; context canceled; EOF Dec 16 03:12:21.014809482 (251h43m20.885892527s) [noise=true , address=77.90.31.10:4520 , piece_id=Q55PSEFHSAOGY7GD4DWI2G56U66BXVZHXZ52DX4LYV4B52F5ZZ6A, seg_pos={0 0}, upload=220713]: Long tail cancellation due to optimal threshold Dec 16 03:12:21.014826394 (251h43m20.885909429s) [seg_pos={0 0}, upload=220713]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 03:12:21.015234772 (251h43m20.886317807s) [upload=220713]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 03:12:21.015241204 (251h43m20.886324249s) [upload=220713]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 03:12:21.319403538 (251h43m21.190486583s) [upload=220713]: Done uploading segments... Dec 16 03:12:21.319427222 (251h43m21.190510257s) [upload=220713]: Upload finished. err: <nil> Duration: 5.871566842s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 6.271633791s trace: https://jaeger.dev.storj.io/trace/2e0c5260bf6208b3 FAIL (2025-12-16 03:20:58 UTC) orbiter warning: context deadline exceeded Duration: 6.27166487s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 5.865387615s trace: https://jaeger.dev.storj.io/trace/13f785634570d9fc FAIL (2025-12-16 03:20:40 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 16 03:12:18.507971969 (251h43m18.379055014s) [upload=220713]: Starting upload Dec 16 03:12:18.508025069 (251h43m18.379108115s) [upload=220713]: Done starting upload Dec 16 03:12:18.508197317 (251h43m18.379280353s) [upload=220713]: Uploading segments... Dec 16 03:12:18.508315022 (251h43m18.379398067s) [upload=220713]: Got next segment. Inline: false Dec 16 03:12:18.509349181 (251h43m18.380432216s) [upload=220713]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 03:12:18.509360643 (251h43m18.380443688s) [upload=220713]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 03:12:18.761879946 (251h43m18.632962991s) [seg_pos={0 0}, upload=220713]: Begin upload segment... Dec 16 03:12:18.761929029 (251h43m18.633012064s) [seg_pos={0 0}, upload=220713]: Done begin upload segment. Dec 16 03:12:18.761967552 (251h43m18.633050597s) [noise=true , address=77.90.31.10:4520 , piece_id=Q55PSEFHSAOGY7GD4DWI2G56U66BXVZHXZ52DX4LYV4B52F5ZZ6A, seg_pos={0 0}, upload=220713]: Uploading piece... Dec 16 03:12:18.762560491 (251h43m18.633643536s) [noise=true , address=45.91.116.6:4974 , piece_id=GQDHG4LFMNUHAKFA3GA3WANCW5OGHOWCYM3IZCUGTWI54LGFWDDA, seg_pos={0 0}, upload=220713]: Uploading piece... Dec 16 03:12:18.762660291 (251h43m18.633743336s) [noise=true , address=145.79.227.25:4617 , piece_id=TCLE2SK75NCWPLGUPYLJI3M5WD75Z2APQUGUXYJFFYAH5MU42T6A, seg_pos={0 0}, upload=220713]: Uploading piece... Dec 16 03:12:18.763036347 (251h43m18.634119382s) [noise=true , address=145.79.227.36:4518 , piece_id=I5C4LFPXUEB2JBUTWHXQV7VRJCCNN2RP7Z3CQ7RQENNMUBNNMHYQ, seg_pos={0 0}, upload=220713]: Uploading piece... Dec 16 03:12:18.763318324 (251h43m18.634401359s) [noise=true , address=145.79.227.14:4840 , piece_id=3ABV6MR5KUUYYF7I2TU5MEXF5ULEY6IU6YIF72DCGR5MBALRWF3A, seg_pos={0 0}, upload=220713]: Uploading piece... Dec 16 03:12:18.763607314 (251h43m18.634690349s) [noise=true , address=145.79.227.50:4373 , piece_id=YMG7Q5VQQKAYHO2HAEPOBRRDARQ25HGALNK3YOMIXUXYQONNKXSA, seg_pos={0 0}, upload=220713]: Uploading piece... Dec 16 03:12:18.763864645 (251h43m18.63494769s) [noise=true , address=145.79.227.51:4409 , piece_id=3IEX2COS67O2QHKMMPC2NZJ3PGDRDG3O5WOKIY5IA4HKMZFDNF4Q, seg_pos={0 0}, upload=220713]: Uploading piece... Dec 16 03:12:18.763888079 (251h43m18.634971154s) [noise=true , address=145.79.227.42:4123 , piece_id=NMJHDWWFESF3GM5TMEZFQGFD7YY5QHIIYQVHF3JIINSJHSEQDFZQ, seg_pos={0 0}, upload=220713]: Uploading piece... Dec 16 03:12:18.763947542 (251h43m18.635030607s) [noise=true , address=145.79.227.32:4834 , piece_id=RRMTYRQ37P6YWUV6BPVUC2BFLCRPWN6Q7VEVX6BDA54IKWTFH7SQ, seg_pos={0 0}, upload=220713]: Uploading piece... Dec 16 03:12:18.764263093 (251h43m18.635346138s) [noise=true , address=145.79.227.37:4565 , piece_id=4VFLWAOYZ7OOHBEDWSBLAWDZODVQEU3MFCQKSNGUE6HLC3MDPVCQ, seg_pos={0 0}, upload=220713]: Uploading piece... Dec 16 03:12:18.764390716 (251h43m18.635473761s) [noise=true , address=145.79.227.44:4194 , piece_id=RJPVGEHG2XIR5JAPDFN6FR5ZH7ICFU2ZQIQKURWSLTBQT4UV7POA, seg_pos={0 0}, upload=220713]: Uploading piece... Dec 16 03:12:18.764941455 (251h43m18.6360245s) [noise=true , address=145.79.227.33:4870 , piece_id=MRUDYCE63CU2SHIFP64BNHFRLYAO2A3KXN6GIZGG7QJGX6S7UXDA, seg_pos={0 0}, upload=220713]: Uploading piece... Dec 16 03:12:18.765504287 (251h43m18.636587322s) [noise=true , address=45.91.116.5:4298 , piece_id=GVZCYRIPTXYE6WOPQPRYKJSGV23I3GZ22SOCOJSXFUBLNDVQN2MA, seg_pos={0 0}, upload=220713]: Uploading piece... Dec 16 03:12:18.766127143 (251h43m18.637210188s) [noise=true , address=145.79.227.29:4942 , piece_id=SON2DHJWJHPLZ7IJG6HFLK6D6C5O6JL2DMLLPOB5GHSPET544HAA, seg_pos={0 0}, upload=220713]: Uploading piece... Dec 16 03:12:18.766432894 (251h43m18.637515929s) [noise=true , address=77.90.31.16:4735 , piece_id=ATZ3AJVXFRAXDM4DR6MKDEHTRHM6AG4RB4YLKDHDLDMO5XFF3MPA, seg_pos={0 0}, upload=220713]: Uploading piece... Dec 16 03:12:18.766736152 (251h43m18.637819187s) [noise=true , address=145.79.227.52:4447 , piece_id=OYGRYLI43OEMB7MBUTGLXG76U4WUUR3EMNCUZQ5C67DVHLWBVHJA, seg_pos={0 0}, upload=220713]: Uploading piece... Dec 16 03:12:20.470095008 (251h43m20.341178064s) [upload=220713]: Next returned nil segment Dec 16 03:12:20.470146787 (251h43m20.341229832s) [upload=220713]: Waiting for error group managing segments... Dec 16 03:12:20.753665772 (251h43m20.624748807s) [noise=true , address=145.79.227.42:4123 , piece_id=NMJHDWWFESF3GM5TMEZFQGFD7YY5QHIIYQVHF3JIINSJHSEQDFZQ, seg_pos={0 0}, upload=220713]: Done uploading piece. err: <nil> Dec 16 03:12:20.757729490 (251h43m20.628812535s) [noise=true , address=145.79.227.29:4942 , piece_id=SON2DHJWJHPLZ7IJG6HFLK6D6C5O6JL2DMLLPOB5GHSPET544HAA, seg_pos={0 0}, upload=220713]: Done uploading piece. err: <nil> Dec 16 03:12:20.760602501 (251h43m20.631685546s) [noise=true , address=145.79.227.36:4518 , piece_id=I5C4LFPXUEB2JBUTWHXQV7VRJCCNN2RP7Z3CQ7RQENNMUBNNMHYQ, seg_pos={0 0}, upload=220713]: Done uploading piece. err: <nil> Dec 16 03:12:20.764326061 (251h43m20.635409106s) [noise=true , address=145.79.227.25:4617 , piece_id=TCLE2SK75NCWPLGUPYLJI3M5WD75Z2APQUGUXYJFFYAH5MU42T6A, seg_pos={0 0}, upload=220713]: Done uploading piece. err: <nil> Dec 16 03:12:20.766406653 (251h43m20.637489698s) [noise=true , address=145.79.227.37:4565 , piece_id=4VFLWAOYZ7OOHBEDWSBLAWDZODVQEU3MFCQKSNGUE6HLC3MDPVCQ, seg_pos={0 0}, upload=220713]: Done uploading piece. err: <nil> Dec 16 03:12:20.770108361 (251h43m20.641191406s) [noise=true , address=145.79.227.33:4870 , piece_id=MRUDYCE63CU2SHIFP64BNHFRLYAO2A3KXN6GIZGG7QJGX6S7UXDA, seg_pos={0 0}, upload=220713]: Done uploading piece. err: <nil> Dec 16 03:12:20.773437491 (251h43m20.644520546s) [noise=true , address=145.79.227.50:4373 , piece_id=YMG7Q5VQQKAYHO2HAEPOBRRDARQ25HGALNK3YOMIXUXYQONNKXSA, seg_pos={0 0}, upload=220713]: Done uploading piece. err: <nil> Dec 16 03:12:20.775082453 (251h43m20.646165488s) [noise=true , address=145.79.227.52:4447 , piece_id=OYGRYLI43OEMB7MBUTGLXG76U4WUUR3EMNCUZQ5C67DVHLWBVHJA, seg_pos={0 0}, upload=220713]: Done uploading piece. err: <nil> Dec 16 03:12:20.833660625 (251h43m20.70474366s) [noise=true , address=145.79.227.51:4409 , piece_id=3IEX2COS67O2QHKMMPC2NZJ3PGDRDG3O5WOKIY5IA4HKMZFDNF4Q, seg_pos={0 0}, upload=220713]: Done uploading piece. err: <nil> Dec 16 03:12:20.844657545 (251h43m20.71574058s) [noise=true , address=145.79.227.14:4840 , piece_id=3ABV6MR5KUUYYF7I2TU5MEXF5ULEY6IU6YIF72DCGR5MBALRWF3A, seg_pos={0 0}, upload=220713]: Done uploading piece. err: <nil> Dec 16 03:12:20.859069218 (251h43m20.730152273s) [noise=true , address=145.79.227.32:4834 , piece_id=RRMTYRQ37P6YWUV6BPVUC2BFLCRPWN6Q7VEVX6BDA54IKWTFH7SQ, seg_pos={0 0}, upload=220713]: Done uploading piece. err: <nil> Dec 16 03:12:21.007293480 (251h43m20.878376515s) [noise=true , address=77.90.31.16:4735 , piece_id=ATZ3AJVXFRAXDM4DR6MKDEHTRHM6AG4RB4YLKDHDLDMO5XFF3MPA, seg_pos={0 0}, upload=220713]: Done uploading piece. err: <nil> Dec 16 03:12:21.014381907 (251h43m20.885464952s) [noise=true , address=45.91.116.5:4298 , piece_id=GVZCYRIPTXYE6WOPQPRYKJSGV23I3GZ22SOCOJSXFUBLNDVQN2MA, seg_pos={0 0}, upload=220713]: Done uploading piece. err: <nil> Dec 16 03:12:21.014403138 (251h43m20.885486173s) [seg_pos={0 0}, upload=220713]: Segment reached optimal threshold of 13 pieces. Dec 16 03:12:21.014711706 (251h43m20.885794741s) [noise=true , address=145.79.227.44:4194 , piece_id=RJPVGEHG2XIR5JAPDFN6FR5ZH7ICFU2ZQIQKURWSLTBQT4UV7POA, seg_pos={0 0}, upload=220713]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh): protocol: expected piece hash; context canceled; EOF Dec 16 03:12:21.014721354 (251h43m20.885804389s) [noise=true , address=145.79.227.44:4194 , piece_id=RJPVGEHG2XIR5JAPDFN6FR5ZH7ICFU2ZQIQKURWSLTBQT4UV7POA, seg_pos={0 0}, upload=220713]: Long tail cancellation due to optimal threshold Dec 16 03:12:21.014764125 (251h43m20.88584716s) [noise=true , address=45.91.116.6:4974 , piece_id=GQDHG4LFMNUHAKFA3GA3WANCW5OGHOWCYM3IZCUGTWI54LGFWDDA, seg_pos={0 0}, upload=220713]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: expected piece hash; context canceled; EOF Dec 16 03:12:21.014782741 (251h43m20.885865786s) [noise=true , address=45.91.116.6:4974 , piece_id=GQDHG4LFMNUHAKFA3GA3WANCW5OGHOWCYM3IZCUGTWI54LGFWDDA, seg_pos={0 0}, upload=220713]: Long tail cancellation due to optimal threshold Dec 16 03:12:21.014804422 (251h43m20.885887467s) [noise=true , address=77.90.31.10:4520 , piece_id=Q55PSEFHSAOGY7GD4DWI2G56U66BXVZHXZ52DX4LYV4B52F5ZZ6A, seg_pos={0 0}, upload=220713]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp): protocol: expected piece hash; context canceled; EOF Dec 16 03:12:21.014809482 (251h43m20.885892527s) [noise=true , address=77.90.31.10:4520 , piece_id=Q55PSEFHSAOGY7GD4DWI2G56U66BXVZHXZ52DX4LYV4B52F5ZZ6A, seg_pos={0 0}, upload=220713]: Long tail cancellation due to optimal threshold Dec 16 03:12:21.014826394 (251h43m20.885909429s) [seg_pos={0 0}, upload=220713]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 03:12:21.015234772 (251h43m20.886317807s) [upload=220713]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 03:12:21.015241204 (251h43m20.886324249s) [upload=220713]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 03:12:21.319403538 (251h43m21.190486583s) [upload=220713]: Done uploading segments... Dec 16 03:12:21.319427222 (251h43m21.190510257s) [upload=220713]: Upload finished. err: <nil> Duration: 5.865416869s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 6.85194038s trace: https://jaeger.dev.storj.io/trace/2e0c5260bf6208b3 FAIL (2025-12-16 03:19:49 UTC) orbiter warning: context deadline exceeded Duration: 6.851957752s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 6.985270803s trace: https://jaeger.dev.storj.io/trace/13f785634570d9fc FAIL (2025-12-16 03:19:42 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 16 03:12:18.507971969 (251h43m18.379055014s) [upload=220713]: Starting upload Dec 16 03:12:18.508025069 (251h43m18.379108115s) [upload=220713]: Done starting upload Dec 16 03:12:18.508197317 (251h43m18.379280353s) [upload=220713]: Uploading segments... Dec 16 03:12:18.508315022 (251h43m18.379398067s) [upload=220713]: Got next segment. Inline: false Dec 16 03:12:18.509349181 (251h43m18.380432216s) [upload=220713]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 03:12:18.509360643 (251h43m18.380443688s) [upload=220713]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 03:12:18.761879946 (251h43m18.632962991s) [seg_pos={0 0}, upload=220713]: Begin upload segment... Dec 16 03:12:18.761929029 (251h43m18.633012064s) [seg_pos={0 0}, upload=220713]: Done begin upload segment. Dec 16 03:12:18.761967552 (251h43m18.633050597s) [noise=true , address=77.90.31.10:4520 , piece_id=Q55PSEFHSAOGY7GD4DWI2G56U66BXVZHXZ52DX4LYV4B52F5ZZ6A, seg_pos={0 0}, upload=220713]: Uploading piece... Dec 16 03:12:18.762560491 (251h43m18.633643536s) [noise=true , address=45.91.116.6:4974 , piece_id=GQDHG4LFMNUHAKFA3GA3WANCW5OGHOWCYM3IZCUGTWI54LGFWDDA, seg_pos={0 0}, upload=220713]: Uploading piece... Dec 16 03:12:18.762660291 (251h43m18.633743336s) [noise=true , address=145.79.227.25:4617 , piece_id=TCLE2SK75NCWPLGUPYLJI3M5WD75Z2APQUGUXYJFFYAH5MU42T6A, seg_pos={0 0}, upload=220713]: Uploading piece... Dec 16 03:12:18.763036347 (251h43m18.634119382s) [noise=true , address=145.79.227.36:4518 , piece_id=I5C4LFPXUEB2JBUTWHXQV7VRJCCNN2RP7Z3CQ7RQENNMUBNNMHYQ, seg_pos={0 0}, upload=220713]: Uploading piece... Dec 16 03:12:18.763318324 (251h43m18.634401359s) [noise=true , address=145.79.227.14:4840 , piece_id=3ABV6MR5KUUYYF7I2TU5MEXF5ULEY6IU6YIF72DCGR5MBALRWF3A, seg_pos={0 0}, upload=220713]: Uploading piece... Dec 16 03:12:18.763607314 (251h43m18.634690349s) [noise=true , address=145.79.227.50:4373 , piece_id=YMG7Q5VQQKAYHO2HAEPOBRRDARQ25HGALNK3YOMIXUXYQONNKXSA, seg_pos={0 0}, upload=220713]: Uploading piece... Dec 16 03:12:18.763864645 (251h43m18.63494769s) [noise=true , address=145.79.227.51:4409 , piece_id=3IEX2COS67O2QHKMMPC2NZJ3PGDRDG3O5WOKIY5IA4HKMZFDNF4Q, seg_pos={0 0}, upload=220713]: Uploading piece... Dec 16 03:12:18.763888079 (251h43m18.634971154s) [noise=true , address=145.79.227.42:4123 , piece_id=NMJHDWWFESF3GM5TMEZFQGFD7YY5QHIIYQVHF3JIINSJHSEQDFZQ, seg_pos={0 0}, upload=220713]: Uploading piece... Dec 16 03:12:18.763947542 (251h43m18.635030607s) [noise=true , address=145.79.227.32:4834 , piece_id=RRMTYRQ37P6YWUV6BPVUC2BFLCRPWN6Q7VEVX6BDA54IKWTFH7SQ, seg_pos={0 0}, upload=220713]: Uploading piece... Dec 16 03:12:18.764263093 (251h43m18.635346138s) [noise=true , address=145.79.227.37:4565 , piece_id=4VFLWAOYZ7OOHBEDWSBLAWDZODVQEU3MFCQKSNGUE6HLC3MDPVCQ, seg_pos={0 0}, upload=220713]: Uploading piece... Dec 16 03:12:18.764390716 (251h43m18.635473761s) [noise=true , address=145.79.227.44:4194 , piece_id=RJPVGEHG2XIR5JAPDFN6FR5ZH7ICFU2ZQIQKURWSLTBQT4UV7POA, seg_pos={0 0}, upload=220713]: Uploading piece... Dec 16 03:12:18.764941455 (251h43m18.6360245s) [noise=true , address=145.79.227.33:4870 , piece_id=MRUDYCE63CU2SHIFP64BNHFRLYAO2A3KXN6GIZGG7QJGX6S7UXDA, seg_pos={0 0}, upload=220713]: Uploading piece... Dec 16 03:12:18.765504287 (251h43m18.636587322s) [noise=true , address=45.91.116.5:4298 , piece_id=GVZCYRIPTXYE6WOPQPRYKJSGV23I3GZ22SOCOJSXFUBLNDVQN2MA, seg_pos={0 0}, upload=220713]: Uploading piece... Dec 16 03:12:18.766127143 (251h43m18.637210188s) [noise=true , address=145.79.227.29:4942 , piece_id=SON2DHJWJHPLZ7IJG6HFLK6D6C5O6JL2DMLLPOB5GHSPET544HAA, seg_pos={0 0}, upload=220713]: Uploading piece... Dec 16 03:12:18.766432894 (251h43m18.637515929s) [noise=true , address=77.90.31.16:4735 , piece_id=ATZ3AJVXFRAXDM4DR6MKDEHTRHM6AG4RB4YLKDHDLDMO5XFF3MPA, seg_pos={0 0}, upload=220713]: Uploading piece... Dec 16 03:12:18.766736152 (251h43m18.637819187s) [noise=true , address=145.79.227.52:4447 , piece_id=OYGRYLI43OEMB7MBUTGLXG76U4WUUR3EMNCUZQ5C67DVHLWBVHJA, seg_pos={0 0}, upload=220713]: Uploading piece... Dec 16 03:12:20.470095008 (251h43m20.341178064s) [upload=220713]: Next returned nil segment Dec 16 03:12:20.470146787 (251h43m20.341229832s) [upload=220713]: Waiting for error group managing segments... Dec 16 03:12:20.753665772 (251h43m20.624748807s) [noise=true , address=145.79.227.42:4123 , piece_id=NMJHDWWFESF3GM5TMEZFQGFD7YY5QHIIYQVHF3JIINSJHSEQDFZQ, seg_pos={0 0}, upload=220713]: Done uploading piece. err: <nil> Dec 16 03:12:20.757729490 (251h43m20.628812535s) [noise=true , address=145.79.227.29:4942 , piece_id=SON2DHJWJHPLZ7IJG6HFLK6D6C5O6JL2DMLLPOB5GHSPET544HAA, seg_pos={0 0}, upload=220713]: Done uploading piece. err: <nil> Dec 16 03:12:20.760602501 (251h43m20.631685546s) [noise=true , address=145.79.227.36:4518 , piece_id=I5C4LFPXUEB2JBUTWHXQV7VRJCCNN2RP7Z3CQ7RQENNMUBNNMHYQ, seg_pos={0 0}, upload=220713]: Done uploading piece. err: <nil> Dec 16 03:12:20.764326061 (251h43m20.635409106s) [noise=true , address=145.79.227.25:4617 , piece_id=TCLE2SK75NCWPLGUPYLJI3M5WD75Z2APQUGUXYJFFYAH5MU42T6A, seg_pos={0 0}, upload=220713]: Done uploading piece. err: <nil> Dec 16 03:12:20.766406653 (251h43m20.637489698s) [noise=true , address=145.79.227.37:4565 , piece_id=4VFLWAOYZ7OOHBEDWSBLAWDZODVQEU3MFCQKSNGUE6HLC3MDPVCQ, seg_pos={0 0}, upload=220713]: Done uploading piece. err: <nil> Dec 16 03:12:20.770108361 (251h43m20.641191406s) [noise=true , address=145.79.227.33:4870 , piece_id=MRUDYCE63CU2SHIFP64BNHFRLYAO2A3KXN6GIZGG7QJGX6S7UXDA, seg_pos={0 0}, upload=220713]: Done uploading piece. err: <nil> Dec 16 03:12:20.773437491 (251h43m20.644520546s) [noise=true , address=145.79.227.50:4373 , piece_id=YMG7Q5VQQKAYHO2HAEPOBRRDARQ25HGALNK3YOMIXUXYQONNKXSA, seg_pos={0 0}, upload=220713]: Done uploading piece. err: <nil> Dec 16 03:12:20.775082453 (251h43m20.646165488s) [noise=true , address=145.79.227.52:4447 , piece_id=OYGRYLI43OEMB7MBUTGLXG76U4WUUR3EMNCUZQ5C67DVHLWBVHJA, seg_pos={0 0}, upload=220713]: Done uploading piece. err: <nil> Dec 16 03:12:20.833660625 (251h43m20.70474366s) [noise=true , address=145.79.227.51:4409 , piece_id=3IEX2COS67O2QHKMMPC2NZJ3PGDRDG3O5WOKIY5IA4HKMZFDNF4Q, seg_pos={0 0}, upload=220713]: Done uploading piece. err: <nil> Dec 16 03:12:20.844657545 (251h43m20.71574058s) [noise=true , address=145.79.227.14:4840 , piece_id=3ABV6MR5KUUYYF7I2TU5MEXF5ULEY6IU6YIF72DCGR5MBALRWF3A, seg_pos={0 0}, upload=220713]: Done uploading piece. err: <nil> Dec 16 03:12:20.859069218 (251h43m20.730152273s) [noise=true , address=145.79.227.32:4834 , piece_id=RRMTYRQ37P6YWUV6BPVUC2BFLCRPWN6Q7VEVX6BDA54IKWTFH7SQ, seg_pos={0 0}, upload=220713]: Done uploading piece. err: <nil> Dec 16 03:12:21.007293480 (251h43m20.878376515s) [noise=true , address=77.90.31.16:4735 , piece_id=ATZ3AJVXFRAXDM4DR6MKDEHTRHM6AG4RB4YLKDHDLDMO5XFF3MPA, seg_pos={0 0}, upload=220713]: Done uploading piece. err: <nil> Dec 16 03:12:21.014381907 (251h43m20.885464952s) [noise=true , address=45.91.116.5:4298 , piece_id=GVZCYRIPTXYE6WOPQPRYKJSGV23I3GZ22SOCOJSXFUBLNDVQN2MA, seg_pos={0 0}, upload=220713]: Done uploading piece. err: <nil> Dec 16 03:12:21.014403138 (251h43m20.885486173s) [seg_pos={0 0}, upload=220713]: Segment reached optimal threshold of 13 pieces. Dec 16 03:12:21.014711706 (251h43m20.885794741s) [noise=true , address=145.79.227.44:4194 , piece_id=RJPVGEHG2XIR5JAPDFN6FR5ZH7ICFU2ZQIQKURWSLTBQT4UV7POA, seg_pos={0 0}, upload=220713]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh): protocol: expected piece hash; context canceled; EOF Dec 16 03:12:21.014721354 (251h43m20.885804389s) [noise=true , address=145.79.227.44:4194 , piece_id=RJPVGEHG2XIR5JAPDFN6FR5ZH7ICFU2ZQIQKURWSLTBQT4UV7POA, seg_pos={0 0}, upload=220713]: Long tail cancellation due to optimal threshold Dec 16 03:12:21.014764125 (251h43m20.88584716s) [noise=true , address=45.91.116.6:4974 , piece_id=GQDHG4LFMNUHAKFA3GA3WANCW5OGHOWCYM3IZCUGTWI54LGFWDDA, seg_pos={0 0}, upload=220713]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: expected piece hash; context canceled; EOF Dec 16 03:12:21.014782741 (251h43m20.885865786s) [noise=true , address=45.91.116.6:4974 , piece_id=GQDHG4LFMNUHAKFA3GA3WANCW5OGHOWCYM3IZCUGTWI54LGFWDDA, seg_pos={0 0}, upload=220713]: Long tail cancellation due to optimal threshold Dec 16 03:12:21.014804422 (251h43m20.885887467s) [noise=true , address=77.90.31.10:4520 , piece_id=Q55PSEFHSAOGY7GD4DWI2G56U66BXVZHXZ52DX4LYV4B52F5ZZ6A, seg_pos={0 0}, upload=220713]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp): protocol: expected piece hash; context canceled; EOF Dec 16 03:12:21.014809482 (251h43m20.885892527s) [noise=true , address=77.90.31.10:4520 , piece_id=Q55PSEFHSAOGY7GD4DWI2G56U66BXVZHXZ52DX4LYV4B52F5ZZ6A, seg_pos={0 0}, upload=220713]: Long tail cancellation due to optimal threshold Dec 16 03:12:21.014826394 (251h43m20.885909429s) [seg_pos={0 0}, upload=220713]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 03:12:21.015234772 (251h43m20.886317807s) [upload=220713]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 03:12:21.015241204 (251h43m20.886324249s) [upload=220713]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 03:12:21.319403538 (251h43m21.190486583s) [upload=220713]: Done uploading segments... Dec 16 03:12:21.319427222 (251h43m21.190510257s) [upload=220713]: Upload finished. err: <nil> Duration: 6.985303023s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 5.824868298s trace: https://jaeger.dev.storj.io/trace/2e0c5260bf6208b3 FAIL (2025-12-16 03:18:48 UTC) orbiter warning: context deadline exceeded Duration: 5.824897092s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 5.912993084s trace: https://jaeger.dev.storj.io/trace/13f785634570d9fc FAIL (2025-12-16 03:18:36 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 16 03:12:18.507971969 (251h43m18.379055014s) [upload=220713]: Starting upload Dec 16 03:12:18.508025069 (251h43m18.379108115s) [upload=220713]: Done starting upload Dec 16 03:12:18.508197317 (251h43m18.379280353s) [upload=220713]: Uploading segments... Dec 16 03:12:18.508315022 (251h43m18.379398067s) [upload=220713]: Got next segment. Inline: false Dec 16 03:12:18.509349181 (251h43m18.380432216s) [upload=220713]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 03:12:18.509360643 (251h43m18.380443688s) [upload=220713]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 03:12:18.761879946 (251h43m18.632962991s) [seg_pos={0 0}, upload=220713]: Begin upload segment... Dec 16 03:12:18.761929029 (251h43m18.633012064s) [seg_pos={0 0}, upload=220713]: Done begin upload segment. Dec 16 03:12:18.761967552 (251h43m18.633050597s) [noise=true , address=77.90.31.10:4520 , piece_id=Q55PSEFHSAOGY7GD4DWI2G56U66BXVZHXZ52DX4LYV4B52F5ZZ6A, seg_pos={0 0}, upload=220713]: Uploading piece... Dec 16 03:12:18.762560491 (251h43m18.633643536s) [noise=true , address=45.91.116.6:4974 , piece_id=GQDHG4LFMNUHAKFA3GA3WANCW5OGHOWCYM3IZCUGTWI54LGFWDDA, seg_pos={0 0}, upload=220713]: Uploading piece... Dec 16 03:12:18.762660291 (251h43m18.633743336s) [noise=true , address=145.79.227.25:4617 , piece_id=TCLE2SK75NCWPLGUPYLJI3M5WD75Z2APQUGUXYJFFYAH5MU42T6A, seg_pos={0 0}, upload=220713]: Uploading piece... Dec 16 03:12:18.763036347 (251h43m18.634119382s) [noise=true , address=145.79.227.36:4518 , piece_id=I5C4LFPXUEB2JBUTWHXQV7VRJCCNN2RP7Z3CQ7RQENNMUBNNMHYQ, seg_pos={0 0}, upload=220713]: Uploading piece... Dec 16 03:12:18.763318324 (251h43m18.634401359s) [noise=true , address=145.79.227.14:4840 , piece_id=3ABV6MR5KUUYYF7I2TU5MEXF5ULEY6IU6YIF72DCGR5MBALRWF3A, seg_pos={0 0}, upload=220713]: Uploading piece... Dec 16 03:12:18.763607314 (251h43m18.634690349s) [noise=true , address=145.79.227.50:4373 , piece_id=YMG7Q5VQQKAYHO2HAEPOBRRDARQ25HGALNK3YOMIXUXYQONNKXSA, seg_pos={0 0}, upload=220713]: Uploading piece... Dec 16 03:12:18.763864645 (251h43m18.63494769s) [noise=true , address=145.79.227.51:4409 , piece_id=3IEX2COS67O2QHKMMPC2NZJ3PGDRDG3O5WOKIY5IA4HKMZFDNF4Q, seg_pos={0 0}, upload=220713]: Uploading piece... Dec 16 03:12:18.763888079 (251h43m18.634971154s) [noise=true , address=145.79.227.42:4123 , piece_id=NMJHDWWFESF3GM5TMEZFQGFD7YY5QHIIYQVHF3JIINSJHSEQDFZQ, seg_pos={0 0}, upload=220713]: Uploading piece... Dec 16 03:12:18.763947542 (251h43m18.635030607s) [noise=true , address=145.79.227.32:4834 , piece_id=RRMTYRQ37P6YWUV6BPVUC2BFLCRPWN6Q7VEVX6BDA54IKWTFH7SQ, seg_pos={0 0}, upload=220713]: Uploading piece... Dec 16 03:12:18.764263093 (251h43m18.635346138s) [noise=true , address=145.79.227.37:4565 , piece_id=4VFLWAOYZ7OOHBEDWSBLAWDZODVQEU3MFCQKSNGUE6HLC3MDPVCQ, seg_pos={0 0}, upload=220713]: Uploading piece... Dec 16 03:12:18.764390716 (251h43m18.635473761s) [noise=true , address=145.79.227.44:4194 , piece_id=RJPVGEHG2XIR5JAPDFN6FR5ZH7ICFU2ZQIQKURWSLTBQT4UV7POA, seg_pos={0 0}, upload=220713]: Uploading piece... Dec 16 03:12:18.764941455 (251h43m18.6360245s) [noise=true , address=145.79.227.33:4870 , piece_id=MRUDYCE63CU2SHIFP64BNHFRLYAO2A3KXN6GIZGG7QJGX6S7UXDA, seg_pos={0 0}, upload=220713]: Uploading piece... Dec 16 03:12:18.765504287 (251h43m18.636587322s) [noise=true , address=45.91.116.5:4298 , piece_id=GVZCYRIPTXYE6WOPQPRYKJSGV23I3GZ22SOCOJSXFUBLNDVQN2MA, seg_pos={0 0}, upload=220713]: Uploading piece... Dec 16 03:12:18.766127143 (251h43m18.637210188s) [noise=true , address=145.79.227.29:4942 , piece_id=SON2DHJWJHPLZ7IJG6HFLK6D6C5O6JL2DMLLPOB5GHSPET544HAA, seg_pos={0 0}, upload=220713]: Uploading piece... Dec 16 03:12:18.766432894 (251h43m18.637515929s) [noise=true , address=77.90.31.16:4735 , piece_id=ATZ3AJVXFRAXDM4DR6MKDEHTRHM6AG4RB4YLKDHDLDMO5XFF3MPA, seg_pos={0 0}, upload=220713]: Uploading piece... Dec 16 03:12:18.766736152 (251h43m18.637819187s) [noise=true , address=145.79.227.52:4447 , piece_id=OYGRYLI43OEMB7MBUTGLXG76U4WUUR3EMNCUZQ5C67DVHLWBVHJA, seg_pos={0 0}, upload=220713]: Uploading piece... Dec 16 03:12:20.470095008 (251h43m20.341178064s) [upload=220713]: Next returned nil segment Dec 16 03:12:20.470146787 (251h43m20.341229832s) [upload=220713]: Waiting for error group managing segments... Dec 16 03:12:20.753665772 (251h43m20.624748807s) [noise=true , address=145.79.227.42:4123 , piece_id=NMJHDWWFESF3GM5TMEZFQGFD7YY5QHIIYQVHF3JIINSJHSEQDFZQ, seg_pos={0 0}, upload=220713]: Done uploading piece. err: <nil> Dec 16 03:12:20.757729490 (251h43m20.628812535s) [noise=true , address=145.79.227.29:4942 , piece_id=SON2DHJWJHPLZ7IJG6HFLK6D6C5O6JL2DMLLPOB5GHSPET544HAA, seg_pos={0 0}, upload=220713]: Done uploading piece. err: <nil> Dec 16 03:12:20.760602501 (251h43m20.631685546s) [noise=true , address=145.79.227.36:4518 , piece_id=I5C4LFPXUEB2JBUTWHXQV7VRJCCNN2RP7Z3CQ7RQENNMUBNNMHYQ, seg_pos={0 0}, upload=220713]: Done uploading piece. err: <nil> Dec 16 03:12:20.764326061 (251h43m20.635409106s) [noise=true , address=145.79.227.25:4617 , piece_id=TCLE2SK75NCWPLGUPYLJI3M5WD75Z2APQUGUXYJFFYAH5MU42T6A, seg_pos={0 0}, upload=220713]: Done uploading piece. err: <nil> Dec 16 03:12:20.766406653 (251h43m20.637489698s) [noise=true , address=145.79.227.37:4565 , piece_id=4VFLWAOYZ7OOHBEDWSBLAWDZODVQEU3MFCQKSNGUE6HLC3MDPVCQ, seg_pos={0 0}, upload=220713]: Done uploading piece. err: <nil> Dec 16 03:12:20.770108361 (251h43m20.641191406s) [noise=true , address=145.79.227.33:4870 , piece_id=MRUDYCE63CU2SHIFP64BNHFRLYAO2A3KXN6GIZGG7QJGX6S7UXDA, seg_pos={0 0}, upload=220713]: Done uploading piece. err: <nil> Dec 16 03:12:20.773437491 (251h43m20.644520546s) [noise=true , address=145.79.227.50:4373 , piece_id=YMG7Q5VQQKAYHO2HAEPOBRRDARQ25HGALNK3YOMIXUXYQONNKXSA, seg_pos={0 0}, upload=220713]: Done uploading piece. err: <nil> Dec 16 03:12:20.775082453 (251h43m20.646165488s) [noise=true , address=145.79.227.52:4447 , piece_id=OYGRYLI43OEMB7MBUTGLXG76U4WUUR3EMNCUZQ5C67DVHLWBVHJA, seg_pos={0 0}, upload=220713]: Done uploading piece. err: <nil> Dec 16 03:12:20.833660625 (251h43m20.70474366s) [noise=true , address=145.79.227.51:4409 , piece_id=3IEX2COS67O2QHKMMPC2NZJ3PGDRDG3O5WOKIY5IA4HKMZFDNF4Q, seg_pos={0 0}, upload=220713]: Done uploading piece. err: <nil> Dec 16 03:12:20.844657545 (251h43m20.71574058s) [noise=true , address=145.79.227.14:4840 , piece_id=3ABV6MR5KUUYYF7I2TU5MEXF5ULEY6IU6YIF72DCGR5MBALRWF3A, seg_pos={0 0}, upload=220713]: Done uploading piece. err: <nil> Dec 16 03:12:20.859069218 (251h43m20.730152273s) [noise=true , address=145.79.227.32:4834 , piece_id=RRMTYRQ37P6YWUV6BPVUC2BFLCRPWN6Q7VEVX6BDA54IKWTFH7SQ, seg_pos={0 0}, upload=220713]: Done uploading piece. err: <nil> Dec 16 03:12:21.007293480 (251h43m20.878376515s) [noise=true , address=77.90.31.16:4735 , piece_id=ATZ3AJVXFRAXDM4DR6MKDEHTRHM6AG4RB4YLKDHDLDMO5XFF3MPA, seg_pos={0 0}, upload=220713]: Done uploading piece. err: <nil> Dec 16 03:12:21.014381907 (251h43m20.885464952s) [noise=true , address=45.91.116.5:4298 , piece_id=GVZCYRIPTXYE6WOPQPRYKJSGV23I3GZ22SOCOJSXFUBLNDVQN2MA, seg_pos={0 0}, upload=220713]: Done uploading piece. err: <nil> Dec 16 03:12:21.014403138 (251h43m20.885486173s) [seg_pos={0 0}, upload=220713]: Segment reached optimal threshold of 13 pieces. Dec 16 03:12:21.014711706 (251h43m20.885794741s) [noise=true , address=145.79.227.44:4194 , piece_id=RJPVGEHG2XIR5JAPDFN6FR5ZH7ICFU2ZQIQKURWSLTBQT4UV7POA, seg_pos={0 0}, upload=220713]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh): protocol: expected piece hash; context canceled; EOF Dec 16 03:12:21.014721354 (251h43m20.885804389s) [noise=true , address=145.79.227.44:4194 , piece_id=RJPVGEHG2XIR5JAPDFN6FR5ZH7ICFU2ZQIQKURWSLTBQT4UV7POA, seg_pos={0 0}, upload=220713]: Long tail cancellation due to optimal threshold Dec 16 03:12:21.014764125 (251h43m20.88584716s) [noise=true , address=45.91.116.6:4974 , piece_id=GQDHG4LFMNUHAKFA3GA3WANCW5OGHOWCYM3IZCUGTWI54LGFWDDA, seg_pos={0 0}, upload=220713]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: expected piece hash; context canceled; EOF Dec 16 03:12:21.014782741 (251h43m20.885865786s) [noise=true , address=45.91.116.6:4974 , piece_id=GQDHG4LFMNUHAKFA3GA3WANCW5OGHOWCYM3IZCUGTWI54LGFWDDA, seg_pos={0 0}, upload=220713]: Long tail cancellation due to optimal threshold Dec 16 03:12:21.014804422 (251h43m20.885887467s) [noise=true , address=77.90.31.10:4520 , piece_id=Q55PSEFHSAOGY7GD4DWI2G56U66BXVZHXZ52DX4LYV4B52F5ZZ6A, seg_pos={0 0}, upload=220713]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp): protocol: expected piece hash; context canceled; EOF Dec 16 03:12:21.014809482 (251h43m20.885892527s) [noise=true , address=77.90.31.10:4520 , piece_id=Q55PSEFHSAOGY7GD4DWI2G56U66BXVZHXZ52DX4LYV4B52F5ZZ6A, seg_pos={0 0}, upload=220713]: Long tail cancellation due to optimal threshold Dec 16 03:12:21.014826394 (251h43m20.885909429s) [seg_pos={0 0}, upload=220713]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 03:12:21.015234772 (251h43m20.886317807s) [upload=220713]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 03:12:21.015241204 (251h43m20.886324249s) [upload=220713]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 03:12:21.319403538 (251h43m21.190486583s) [upload=220713]: Done uploading segments... Dec 16 03:12:21.319427222 (251h43m21.190510257s) [upload=220713]: Upload finished. err: <nil> Duration: 5.913021439s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 6.091715171s trace: https://jaeger.dev.storj.io/trace/2e0c5260bf6208b3 FAIL (2025-12-16 03:17:46 UTC) orbiter warning: context deadline exceeded Duration: 6.091730839s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 5.78942963s trace: https://jaeger.dev.storj.io/trace/13f785634570d9fc FAIL (2025-12-16 03:17:45 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 16 03:12:18.507971969 (251h43m18.379055014s) [upload=220713]: Starting upload Dec 16 03:12:18.508025069 (251h43m18.379108115s) [upload=220713]: Done starting upload Dec 16 03:12:18.508197317 (251h43m18.379280353s) [upload=220713]: Uploading segments... Dec 16 03:12:18.508315022 (251h43m18.379398067s) [upload=220713]: Got next segment. Inline: false Dec 16 03:12:18.509349181 (251h43m18.380432216s) [upload=220713]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 03:12:18.509360643 (251h43m18.380443688s) [upload=220713]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 03:12:18.761879946 (251h43m18.632962991s) [seg_pos={0 0}, upload=220713]: Begin upload segment... Dec 16 03:12:18.761929029 (251h43m18.633012064s) [seg_pos={0 0}, upload=220713]: Done begin upload segment. Dec 16 03:12:18.761967552 (251h43m18.633050597s) [noise=true , address=77.90.31.10:4520 , piece_id=Q55PSEFHSAOGY7GD4DWI2G56U66BXVZHXZ52DX4LYV4B52F5ZZ6A, seg_pos={0 0}, upload=220713]: Uploading piece... Dec 16 03:12:18.762560491 (251h43m18.633643536s) [noise=true , address=45.91.116.6:4974 , piece_id=GQDHG4LFMNUHAKFA3GA3WANCW5OGHOWCYM3IZCUGTWI54LGFWDDA, seg_pos={0 0}, upload=220713]: Uploading piece... Dec 16 03:12:18.762660291 (251h43m18.633743336s) [noise=true , address=145.79.227.25:4617 , piece_id=TCLE2SK75NCWPLGUPYLJI3M5WD75Z2APQUGUXYJFFYAH5MU42T6A, seg_pos={0 0}, upload=220713]: Uploading piece... Dec 16 03:12:18.763036347 (251h43m18.634119382s) [noise=true , address=145.79.227.36:4518 , piece_id=I5C4LFPXUEB2JBUTWHXQV7VRJCCNN2RP7Z3CQ7RQENNMUBNNMHYQ, seg_pos={0 0}, upload=220713]: Uploading piece... Dec 16 03:12:18.763318324 (251h43m18.634401359s) [noise=true , address=145.79.227.14:4840 , piece_id=3ABV6MR5KUUYYF7I2TU5MEXF5ULEY6IU6YIF72DCGR5MBALRWF3A, seg_pos={0 0}, upload=220713]: Uploading piece... Dec 16 03:12:18.763607314 (251h43m18.634690349s) [noise=true , address=145.79.227.50:4373 , piece_id=YMG7Q5VQQKAYHO2HAEPOBRRDARQ25HGALNK3YOMIXUXYQONNKXSA, seg_pos={0 0}, upload=220713]: Uploading piece... Dec 16 03:12:18.763864645 (251h43m18.63494769s) [noise=true , address=145.79.227.51:4409 , piece_id=3IEX2COS67O2QHKMMPC2NZJ3PGDRDG3O5WOKIY5IA4HKMZFDNF4Q, seg_pos={0 0}, upload=220713]: Uploading piece... Dec 16 03:12:18.763888079 (251h43m18.634971154s) [noise=true , address=145.79.227.42:4123 , piece_id=NMJHDWWFESF3GM5TMEZFQGFD7YY5QHIIYQVHF3JIINSJHSEQDFZQ, seg_pos={0 0}, upload=220713]: Uploading piece... Dec 16 03:12:18.763947542 (251h43m18.635030607s) [noise=true , address=145.79.227.32:4834 , piece_id=RRMTYRQ37P6YWUV6BPVUC2BFLCRPWN6Q7VEVX6BDA54IKWTFH7SQ, seg_pos={0 0}, upload=220713]: Uploading piece... Dec 16 03:12:18.764263093 (251h43m18.635346138s) [noise=true , address=145.79.227.37:4565 , piece_id=4VFLWAOYZ7OOHBEDWSBLAWDZODVQEU3MFCQKSNGUE6HLC3MDPVCQ, seg_pos={0 0}, upload=220713]: Uploading piece... Dec 16 03:12:18.764390716 (251h43m18.635473761s) [noise=true , address=145.79.227.44:4194 , piece_id=RJPVGEHG2XIR5JAPDFN6FR5ZH7ICFU2ZQIQKURWSLTBQT4UV7POA, seg_pos={0 0}, upload=220713]: Uploading piece... Dec 16 03:12:18.764941455 (251h43m18.6360245s) [noise=true , address=145.79.227.33:4870 , piece_id=MRUDYCE63CU2SHIFP64BNHFRLYAO2A3KXN6GIZGG7QJGX6S7UXDA, seg_pos={0 0}, upload=220713]: Uploading piece... Dec 16 03:12:18.765504287 (251h43m18.636587322s) [noise=true , address=45.91.116.5:4298 , piece_id=GVZCYRIPTXYE6WOPQPRYKJSGV23I3GZ22SOCOJSXFUBLNDVQN2MA, seg_pos={0 0}, upload=220713]: Uploading piece... Dec 16 03:12:18.766127143 (251h43m18.637210188s) [noise=true , address=145.79.227.29:4942 , piece_id=SON2DHJWJHPLZ7IJG6HFLK6D6C5O6JL2DMLLPOB5GHSPET544HAA, seg_pos={0 0}, upload=220713]: Uploading piece... Dec 16 03:12:18.766432894 (251h43m18.637515929s) [noise=true , address=77.90.31.16:4735 , piece_id=ATZ3AJVXFRAXDM4DR6MKDEHTRHM6AG4RB4YLKDHDLDMO5XFF3MPA, seg_pos={0 0}, upload=220713]: Uploading piece... Dec 16 03:12:18.766736152 (251h43m18.637819187s) [noise=true , address=145.79.227.52:4447 , piece_id=OYGRYLI43OEMB7MBUTGLXG76U4WUUR3EMNCUZQ5C67DVHLWBVHJA, seg_pos={0 0}, upload=220713]: Uploading piece... Dec 16 03:12:20.470095008 (251h43m20.341178064s) [upload=220713]: Next returned nil segment Dec 16 03:12:20.470146787 (251h43m20.341229832s) [upload=220713]: Waiting for error group managing segments... Dec 16 03:12:20.753665772 (251h43m20.624748807s) [noise=true , address=145.79.227.42:4123 , piece_id=NMJHDWWFESF3GM5TMEZFQGFD7YY5QHIIYQVHF3JIINSJHSEQDFZQ, seg_pos={0 0}, upload=220713]: Done uploading piece. err: <nil> Dec 16 03:12:20.757729490 (251h43m20.628812535s) [noise=true , address=145.79.227.29:4942 , piece_id=SON2DHJWJHPLZ7IJG6HFLK6D6C5O6JL2DMLLPOB5GHSPET544HAA, seg_pos={0 0}, upload=220713]: Done uploading piece. err: <nil> Dec 16 03:12:20.760602501 (251h43m20.631685546s) [noise=true , address=145.79.227.36:4518 , piece_id=I5C4LFPXUEB2JBUTWHXQV7VRJCCNN2RP7Z3CQ7RQENNMUBNNMHYQ, seg_pos={0 0}, upload=220713]: Done uploading piece. err: <nil> Dec 16 03:12:20.764326061 (251h43m20.635409106s) [noise=true , address=145.79.227.25:4617 , piece_id=TCLE2SK75NCWPLGUPYLJI3M5WD75Z2APQUGUXYJFFYAH5MU42T6A, seg_pos={0 0}, upload=220713]: Done uploading piece. err: <nil> Dec 16 03:12:20.766406653 (251h43m20.637489698s) [noise=true , address=145.79.227.37:4565 , piece_id=4VFLWAOYZ7OOHBEDWSBLAWDZODVQEU3MFCQKSNGUE6HLC3MDPVCQ, seg_pos={0 0}, upload=220713]: Done uploading piece. err: <nil> Dec 16 03:12:20.770108361 (251h43m20.641191406s) [noise=true , address=145.79.227.33:4870 , piece_id=MRUDYCE63CU2SHIFP64BNHFRLYAO2A3KXN6GIZGG7QJGX6S7UXDA, seg_pos={0 0}, upload=220713]: Done uploading piece. err: <nil> Dec 16 03:12:20.773437491 (251h43m20.644520546s) [noise=true , address=145.79.227.50:4373 , piece_id=YMG7Q5VQQKAYHO2HAEPOBRRDARQ25HGALNK3YOMIXUXYQONNKXSA, seg_pos={0 0}, upload=220713]: Done uploading piece. err: <nil> Dec 16 03:12:20.775082453 (251h43m20.646165488s) [noise=true , address=145.79.227.52:4447 , piece_id=OYGRYLI43OEMB7MBUTGLXG76U4WUUR3EMNCUZQ5C67DVHLWBVHJA, seg_pos={0 0}, upload=220713]: Done uploading piece. err: <nil> Dec 16 03:12:20.833660625 (251h43m20.70474366s) [noise=true , address=145.79.227.51:4409 , piece_id=3IEX2COS67O2QHKMMPC2NZJ3PGDRDG3O5WOKIY5IA4HKMZFDNF4Q, seg_pos={0 0}, upload=220713]: Done uploading piece. err: <nil> Dec 16 03:12:20.844657545 (251h43m20.71574058s) [noise=true , address=145.79.227.14:4840 , piece_id=3ABV6MR5KUUYYF7I2TU5MEXF5ULEY6IU6YIF72DCGR5MBALRWF3A, seg_pos={0 0}, upload=220713]: Done uploading piece. err: <nil> Dec 16 03:12:20.859069218 (251h43m20.730152273s) [noise=true , address=145.79.227.32:4834 , piece_id=RRMTYRQ37P6YWUV6BPVUC2BFLCRPWN6Q7VEVX6BDA54IKWTFH7SQ, seg_pos={0 0}, upload=220713]: Done uploading piece. err: <nil> Dec 16 03:12:21.007293480 (251h43m20.878376515s) [noise=true , address=77.90.31.16:4735 , piece_id=ATZ3AJVXFRAXDM4DR6MKDEHTRHM6AG4RB4YLKDHDLDMO5XFF3MPA, seg_pos={0 0}, upload=220713]: Done uploading piece. err: <nil> Dec 16 03:12:21.014381907 (251h43m20.885464952s) [noise=true , address=45.91.116.5:4298 , piece_id=GVZCYRIPTXYE6WOPQPRYKJSGV23I3GZ22SOCOJSXFUBLNDVQN2MA, seg_pos={0 0}, upload=220713]: Done uploading piece. err: <nil> Dec 16 03:12:21.014403138 (251h43m20.885486173s) [seg_pos={0 0}, upload=220713]: Segment reached optimal threshold of 13 pieces. Dec 16 03:12:21.014711706 (251h43m20.885794741s) [noise=true , address=145.79.227.44:4194 , piece_id=RJPVGEHG2XIR5JAPDFN6FR5ZH7ICFU2ZQIQKURWSLTBQT4UV7POA, seg_pos={0 0}, upload=220713]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh): protocol: expected piece hash; context canceled; EOF Dec 16 03:12:21.014721354 (251h43m20.885804389s) [noise=true , address=145.79.227.44:4194 , piece_id=RJPVGEHG2XIR5JAPDFN6FR5ZH7ICFU2ZQIQKURWSLTBQT4UV7POA, seg_pos={0 0}, upload=220713]: Long tail cancellation due to optimal threshold Dec 16 03:12:21.014764125 (251h43m20.88584716s) [noise=true , address=45.91.116.6:4974 , piece_id=GQDHG4LFMNUHAKFA3GA3WANCW5OGHOWCYM3IZCUGTWI54LGFWDDA, seg_pos={0 0}, upload=220713]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: expected piece hash; context canceled; EOF Dec 16 03:12:21.014782741 (251h43m20.885865786s) [noise=true , address=45.91.116.6:4974 , piece_id=GQDHG4LFMNUHAKFA3GA3WANCW5OGHOWCYM3IZCUGTWI54LGFWDDA, seg_pos={0 0}, upload=220713]: Long tail cancellation due to optimal threshold Dec 16 03:12:21.014804422 (251h43m20.885887467s) [noise=true , address=77.90.31.10:4520 , piece_id=Q55PSEFHSAOGY7GD4DWI2G56U66BXVZHXZ52DX4LYV4B52F5ZZ6A, seg_pos={0 0}, upload=220713]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp): protocol: expected piece hash; context canceled; EOF Dec 16 03:12:21.014809482 (251h43m20.885892527s) [noise=true , address=77.90.31.10:4520 , piece_id=Q55PSEFHSAOGY7GD4DWI2G56U66BXVZHXZ52DX4LYV4B52F5ZZ6A, seg_pos={0 0}, upload=220713]: Long tail cancellation due to optimal threshold Dec 16 03:12:21.014826394 (251h43m20.885909429s) [seg_pos={0 0}, upload=220713]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 03:12:21.015234772 (251h43m20.886317807s) [upload=220713]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 03:12:21.015241204 (251h43m20.886324249s) [upload=220713]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 03:12:21.319403538 (251h43m21.190486583s) [upload=220713]: Done uploading segments... Dec 16 03:12:21.319427222 (251h43m21.190510257s) [upload=220713]: Upload finished. err: <nil> Duration: 5.789456922s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 6.137146466s trace: https://jaeger.dev.storj.io/trace/2e0c5260bf6208b3 FAIL (2025-12-16 03:17:20 UTC) orbiter warning: context deadline exceeded Duration: 6.137175662s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 5.894431246s trace: https://jaeger.dev.storj.io/trace/13f785634570d9fc FAIL (2025-12-16 03:16:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 16 03:12:18.507971969 (251h43m18.379055014s) [upload=220713]: Starting upload Dec 16 03:12:18.508025069 (251h43m18.379108115s) [upload=220713]: Done starting upload Dec 16 03:12:18.508197317 (251h43m18.379280353s) [upload=220713]: Uploading segments... Dec 16 03:12:18.508315022 (251h43m18.379398067s) [upload=220713]: Got next segment. Inline: false Dec 16 03:12:18.509349181 (251h43m18.380432216s) [upload=220713]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 03:12:18.509360643 (251h43m18.380443688s) [upload=220713]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 03:12:18.761879946 (251h43m18.632962991s) [seg_pos={0 0}, upload=220713]: Begin upload segment... Dec 16 03:12:18.761929029 (251h43m18.633012064s) [seg_pos={0 0}, upload=220713]: Done begin upload segment. Dec 16 03:12:18.761967552 (251h43m18.633050597s) [noise=true , address=77.90.31.10:4520 , piece_id=Q55PSEFHSAOGY7GD4DWI2G56U66BXVZHXZ52DX4LYV4B52F5ZZ6A, seg_pos={0 0}, upload=220713]: Uploading piece... Dec 16 03:12:18.762560491 (251h43m18.633643536s) [noise=true , address=45.91.116.6:4974 , piece_id=GQDHG4LFMNUHAKFA3GA3WANCW5OGHOWCYM3IZCUGTWI54LGFWDDA, seg_pos={0 0}, upload=220713]: Uploading piece... Dec 16 03:12:18.762660291 (251h43m18.633743336s) [noise=true , address=145.79.227.25:4617 , piece_id=TCLE2SK75NCWPLGUPYLJI3M5WD75Z2APQUGUXYJFFYAH5MU42T6A, seg_pos={0 0}, upload=220713]: Uploading piece... Dec 16 03:12:18.763036347 (251h43m18.634119382s) [noise=true , address=145.79.227.36:4518 , piece_id=I5C4LFPXUEB2JBUTWHXQV7VRJCCNN2RP7Z3CQ7RQENNMUBNNMHYQ, seg_pos={0 0}, upload=220713]: Uploading piece... Dec 16 03:12:18.763318324 (251h43m18.634401359s) [noise=true , address=145.79.227.14:4840 , piece_id=3ABV6MR5KUUYYF7I2TU5MEXF5ULEY6IU6YIF72DCGR5MBALRWF3A, seg_pos={0 0}, upload=220713]: Uploading piece... Dec 16 03:12:18.763607314 (251h43m18.634690349s) [noise=true , address=145.79.227.50:4373 , piece_id=YMG7Q5VQQKAYHO2HAEPOBRRDARQ25HGALNK3YOMIXUXYQONNKXSA, seg_pos={0 0}, upload=220713]: Uploading piece... Dec 16 03:12:18.763864645 (251h43m18.63494769s) [noise=true , address=145.79.227.51:4409 , piece_id=3IEX2COS67O2QHKMMPC2NZJ3PGDRDG3O5WOKIY5IA4HKMZFDNF4Q, seg_pos={0 0}, upload=220713]: Uploading piece... Dec 16 03:12:18.763888079 (251h43m18.634971154s) [noise=true , address=145.79.227.42:4123 , piece_id=NMJHDWWFESF3GM5TMEZFQGFD7YY5QHIIYQVHF3JIINSJHSEQDFZQ, seg_pos={0 0}, upload=220713]: Uploading piece... Dec 16 03:12:18.763947542 (251h43m18.635030607s) [noise=true , address=145.79.227.32:4834 , piece_id=RRMTYRQ37P6YWUV6BPVUC2BFLCRPWN6Q7VEVX6BDA54IKWTFH7SQ, seg_pos={0 0}, upload=220713]: Uploading piece... Dec 16 03:12:18.764263093 (251h43m18.635346138s) [noise=true , address=145.79.227.37:4565 , piece_id=4VFLWAOYZ7OOHBEDWSBLAWDZODVQEU3MFCQKSNGUE6HLC3MDPVCQ, seg_pos={0 0}, upload=220713]: Uploading piece... Dec 16 03:12:18.764390716 (251h43m18.635473761s) [noise=true , address=145.79.227.44:4194 , piece_id=RJPVGEHG2XIR5JAPDFN6FR5ZH7ICFU2ZQIQKURWSLTBQT4UV7POA, seg_pos={0 0}, upload=220713]: Uploading piece... Dec 16 03:12:18.764941455 (251h43m18.6360245s) [noise=true , address=145.79.227.33:4870 , piece_id=MRUDYCE63CU2SHIFP64BNHFRLYAO2A3KXN6GIZGG7QJGX6S7UXDA, seg_pos={0 0}, upload=220713]: Uploading piece... Dec 16 03:12:18.765504287 (251h43m18.636587322s) [noise=true , address=45.91.116.5:4298 , piece_id=GVZCYRIPTXYE6WOPQPRYKJSGV23I3GZ22SOCOJSXFUBLNDVQN2MA, seg_pos={0 0}, upload=220713]: Uploading piece... Dec 16 03:12:18.766127143 (251h43m18.637210188s) [noise=true , address=145.79.227.29:4942 , piece_id=SON2DHJWJHPLZ7IJG6HFLK6D6C5O6JL2DMLLPOB5GHSPET544HAA, seg_pos={0 0}, upload=220713]: Uploading piece... Dec 16 03:12:18.766432894 (251h43m18.637515929s) [noise=true , address=77.90.31.16:4735 , piece_id=ATZ3AJVXFRAXDM4DR6MKDEHTRHM6AG4RB4YLKDHDLDMO5XFF3MPA, seg_pos={0 0}, upload=220713]: Uploading piece... Dec 16 03:12:18.766736152 (251h43m18.637819187s) [noise=true , address=145.79.227.52:4447 , piece_id=OYGRYLI43OEMB7MBUTGLXG76U4WUUR3EMNCUZQ5C67DVHLWBVHJA, seg_pos={0 0}, upload=220713]: Uploading piece... Dec 16 03:12:20.470095008 (251h43m20.341178064s) [upload=220713]: Next returned nil segment Dec 16 03:12:20.470146787 (251h43m20.341229832s) [upload=220713]: Waiting for error group managing segments... Dec 16 03:12:20.753665772 (251h43m20.624748807s) [noise=true , address=145.79.227.42:4123 , piece_id=NMJHDWWFESF3GM5TMEZFQGFD7YY5QHIIYQVHF3JIINSJHSEQDFZQ, seg_pos={0 0}, upload=220713]: Done uploading piece. err: <nil> Dec 16 03:12:20.757729490 (251h43m20.628812535s) [noise=true , address=145.79.227.29:4942 , piece_id=SON2DHJWJHPLZ7IJG6HFLK6D6C5O6JL2DMLLPOB5GHSPET544HAA, seg_pos={0 0}, upload=220713]: Done uploading piece. err: <nil> Dec 16 03:12:20.760602501 (251h43m20.631685546s) [noise=true , address=145.79.227.36:4518 , piece_id=I5C4LFPXUEB2JBUTWHXQV7VRJCCNN2RP7Z3CQ7RQENNMUBNNMHYQ, seg_pos={0 0}, upload=220713]: Done uploading piece. err: <nil> Dec 16 03:12:20.764326061 (251h43m20.635409106s) [noise=true , address=145.79.227.25:4617 , piece_id=TCLE2SK75NCWPLGUPYLJI3M5WD75Z2APQUGUXYJFFYAH5MU42T6A, seg_pos={0 0}, upload=220713]: Done uploading piece. err: <nil> Dec 16 03:12:20.766406653 (251h43m20.637489698s) [noise=true , address=145.79.227.37:4565 , piece_id=4VFLWAOYZ7OOHBEDWSBLAWDZODVQEU3MFCQKSNGUE6HLC3MDPVCQ, seg_pos={0 0}, upload=220713]: Done uploading piece. err: <nil> Dec 16 03:12:20.770108361 (251h43m20.641191406s) [noise=true , address=145.79.227.33:4870 , piece_id=MRUDYCE63CU2SHIFP64BNHFRLYAO2A3KXN6GIZGG7QJGX6S7UXDA, seg_pos={0 0}, upload=220713]: Done uploading piece. err: <nil> Dec 16 03:12:20.773437491 (251h43m20.644520546s) [noise=true , address=145.79.227.50:4373 , piece_id=YMG7Q5VQQKAYHO2HAEPOBRRDARQ25HGALNK3YOMIXUXYQONNKXSA, seg_pos={0 0}, upload=220713]: Done uploading piece. err: <nil> Dec 16 03:12:20.775082453 (251h43m20.646165488s) [noise=true , address=145.79.227.52:4447 , piece_id=OYGRYLI43OEMB7MBUTGLXG76U4WUUR3EMNCUZQ5C67DVHLWBVHJA, seg_pos={0 0}, upload=220713]: Done uploading piece. err: <nil> Dec 16 03:12:20.833660625 (251h43m20.70474366s) [noise=true , address=145.79.227.51:4409 , piece_id=3IEX2COS67O2QHKMMPC2NZJ3PGDRDG3O5WOKIY5IA4HKMZFDNF4Q, seg_pos={0 0}, upload=220713]: Done uploading piece. err: <nil> Dec 16 03:12:20.844657545 (251h43m20.71574058s) [noise=true , address=145.79.227.14:4840 , piece_id=3ABV6MR5KUUYYF7I2TU5MEXF5ULEY6IU6YIF72DCGR5MBALRWF3A, seg_pos={0 0}, upload=220713]: Done uploading piece. err: <nil> Dec 16 03:12:20.859069218 (251h43m20.730152273s) [noise=true , address=145.79.227.32:4834 , piece_id=RRMTYRQ37P6YWUV6BPVUC2BFLCRPWN6Q7VEVX6BDA54IKWTFH7SQ, seg_pos={0 0}, upload=220713]: Done uploading piece. err: <nil> Dec 16 03:12:21.007293480 (251h43m20.878376515s) [noise=true , address=77.90.31.16:4735 , piece_id=ATZ3AJVXFRAXDM4DR6MKDEHTRHM6AG4RB4YLKDHDLDMO5XFF3MPA, seg_pos={0 0}, upload=220713]: Done uploading piece. err: <nil> Dec 16 03:12:21.014381907 (251h43m20.885464952s) [noise=true , address=45.91.116.5:4298 , piece_id=GVZCYRIPTXYE6WOPQPRYKJSGV23I3GZ22SOCOJSXFUBLNDVQN2MA, seg_pos={0 0}, upload=220713]: Done uploading piece. err: <nil> Dec 16 03:12:21.014403138 (251h43m20.885486173s) [seg_pos={0 0}, upload=220713]: Segment reached optimal threshold of 13 pieces. Dec 16 03:12:21.014711706 (251h43m20.885794741s) [noise=true , address=145.79.227.44:4194 , piece_id=RJPVGEHG2XIR5JAPDFN6FR5ZH7ICFU2ZQIQKURWSLTBQT4UV7POA, seg_pos={0 0}, upload=220713]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh): protocol: expected piece hash; context canceled; EOF Dec 16 03:12:21.014721354 (251h43m20.885804389s) [noise=true , address=145.79.227.44:4194 , piece_id=RJPVGEHG2XIR5JAPDFN6FR5ZH7ICFU2ZQIQKURWSLTBQT4UV7POA, seg_pos={0 0}, upload=220713]: Long tail cancellation due to optimal threshold Dec 16 03:12:21.014764125 (251h43m20.88584716s) [noise=true , address=45.91.116.6:4974 , piece_id=GQDHG4LFMNUHAKFA3GA3WANCW5OGHOWCYM3IZCUGTWI54LGFWDDA, seg_pos={0 0}, upload=220713]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: expected piece hash; context canceled; EOF Dec 16 03:12:21.014782741 (251h43m20.885865786s) [noise=true , address=45.91.116.6:4974 , piece_id=GQDHG4LFMNUHAKFA3GA3WANCW5OGHOWCYM3IZCUGTWI54LGFWDDA, seg_pos={0 0}, upload=220713]: Long tail cancellation due to optimal threshold Dec 16 03:12:21.014804422 (251h43m20.885887467s) [noise=true , address=77.90.31.10:4520 , piece_id=Q55PSEFHSAOGY7GD4DWI2G56U66BXVZHXZ52DX4LYV4B52F5ZZ6A, seg_pos={0 0}, upload=220713]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp): protocol: expected piece hash; context canceled; EOF Dec 16 03:12:21.014809482 (251h43m20.885892527s) [noise=true , address=77.90.31.10:4520 , piece_id=Q55PSEFHSAOGY7GD4DWI2G56U66BXVZHXZ52DX4LYV4B52F5ZZ6A, seg_pos={0 0}, upload=220713]: Long tail cancellation due to optimal threshold Dec 16 03:12:21.014826394 (251h43m20.885909429s) [seg_pos={0 0}, upload=220713]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 03:12:21.015234772 (251h43m20.886317807s) [upload=220713]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 03:12:21.015241204 (251h43m20.886324249s) [upload=220713]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 03:12:21.319403538 (251h43m21.190486583s) [upload=220713]: Done uploading segments... Dec 16 03:12:21.319427222 (251h43m21.190510257s) [upload=220713]: Upload finished. err: <nil> Duration: 5.894452205s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 6.076995689s trace: https://jaeger.dev.storj.io/trace/2e0c5260bf6208b3 FAIL (2025-12-16 03:16:37 UTC) orbiter warning: context deadline exceeded Duration: 6.077042348s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 5.874631554s trace: https://jaeger.dev.storj.io/trace/2e0c5260bf6208b3 FAIL (2025-12-16 03:16:10 UTC) orbiter warning: context deadline exceeded Duration: 5.874706618s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 5.978709129s trace: https://jaeger.dev.storj.io/trace/13f785634570d9fc FAIL (2025-12-16 03:16:02 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 16 03:12:18.507971969 (251h43m18.379055014s) [upload=220713]: Starting upload Dec 16 03:12:18.508025069 (251h43m18.379108115s) [upload=220713]: Done starting upload Dec 16 03:12:18.508197317 (251h43m18.379280353s) [upload=220713]: Uploading segments... Dec 16 03:12:18.508315022 (251h43m18.379398067s) [upload=220713]: Got next segment. Inline: false Dec 16 03:12:18.509349181 (251h43m18.380432216s) [upload=220713]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 03:12:18.509360643 (251h43m18.380443688s) [upload=220713]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 03:12:18.761879946 (251h43m18.632962991s) [seg_pos={0 0}, upload=220713]: Begin upload segment... Dec 16 03:12:18.761929029 (251h43m18.633012064s) [seg_pos={0 0}, upload=220713]: Done begin upload segment. Dec 16 03:12:18.761967552 (251h43m18.633050597s) [noise=true , address=77.90.31.10:4520 , piece_id=Q55PSEFHSAOGY7GD4DWI2G56U66BXVZHXZ52DX4LYV4B52F5ZZ6A, seg_pos={0 0}, upload=220713]: Uploading piece... Dec 16 03:12:18.762560491 (251h43m18.633643536s) [noise=true , address=45.91.116.6:4974 , piece_id=GQDHG4LFMNUHAKFA3GA3WANCW5OGHOWCYM3IZCUGTWI54LGFWDDA, seg_pos={0 0}, upload=220713]: Uploading piece... Dec 16 03:12:18.762660291 (251h43m18.633743336s) [noise=true , address=145.79.227.25:4617 , piece_id=TCLE2SK75NCWPLGUPYLJI3M5WD75Z2APQUGUXYJFFYAH5MU42T6A, seg_pos={0 0}, upload=220713]: Uploading piece... Dec 16 03:12:18.763036347 (251h43m18.634119382s) [noise=true , address=145.79.227.36:4518 , piece_id=I5C4LFPXUEB2JBUTWHXQV7VRJCCNN2RP7Z3CQ7RQENNMUBNNMHYQ, seg_pos={0 0}, upload=220713]: Uploading piece... Dec 16 03:12:18.763318324 (251h43m18.634401359s) [noise=true , address=145.79.227.14:4840 , piece_id=3ABV6MR5KUUYYF7I2TU5MEXF5ULEY6IU6YIF72DCGR5MBALRWF3A, seg_pos={0 0}, upload=220713]: Uploading piece... Dec 16 03:12:18.763607314 (251h43m18.634690349s) [noise=true , address=145.79.227.50:4373 , piece_id=YMG7Q5VQQKAYHO2HAEPOBRRDARQ25HGALNK3YOMIXUXYQONNKXSA, seg_pos={0 0}, upload=220713]: Uploading piece... Dec 16 03:12:18.763864645 (251h43m18.63494769s) [noise=true , address=145.79.227.51:4409 , piece_id=3IEX2COS67O2QHKMMPC2NZJ3PGDRDG3O5WOKIY5IA4HKMZFDNF4Q, seg_pos={0 0}, upload=220713]: Uploading piece... Dec 16 03:12:18.763888079 (251h43m18.634971154s) [noise=true , address=145.79.227.42:4123 , piece_id=NMJHDWWFESF3GM5TMEZFQGFD7YY5QHIIYQVHF3JIINSJHSEQDFZQ, seg_pos={0 0}, upload=220713]: Uploading piece... Dec 16 03:12:18.763947542 (251h43m18.635030607s) [noise=true , address=145.79.227.32:4834 , piece_id=RRMTYRQ37P6YWUV6BPVUC2BFLCRPWN6Q7VEVX6BDA54IKWTFH7SQ, seg_pos={0 0}, upload=220713]: Uploading piece... Dec 16 03:12:18.764263093 (251h43m18.635346138s) [noise=true , address=145.79.227.37:4565 , piece_id=4VFLWAOYZ7OOHBEDWSBLAWDZODVQEU3MFCQKSNGUE6HLC3MDPVCQ, seg_pos={0 0}, upload=220713]: Uploading piece... Dec 16 03:12:18.764390716 (251h43m18.635473761s) [noise=true , address=145.79.227.44:4194 , piece_id=RJPVGEHG2XIR5JAPDFN6FR5ZH7ICFU2ZQIQKURWSLTBQT4UV7POA, seg_pos={0 0}, upload=220713]: Uploading piece... Dec 16 03:12:18.764941455 (251h43m18.6360245s) [noise=true , address=145.79.227.33:4870 , piece_id=MRUDYCE63CU2SHIFP64BNHFRLYAO2A3KXN6GIZGG7QJGX6S7UXDA, seg_pos={0 0}, upload=220713]: Uploading piece... Dec 16 03:12:18.765504287 (251h43m18.636587322s) [noise=true , address=45.91.116.5:4298 , piece_id=GVZCYRIPTXYE6WOPQPRYKJSGV23I3GZ22SOCOJSXFUBLNDVQN2MA, seg_pos={0 0}, upload=220713]: Uploading piece... Dec 16 03:12:18.766127143 (251h43m18.637210188s) [noise=true , address=145.79.227.29:4942 , piece_id=SON2DHJWJHPLZ7IJG6HFLK6D6C5O6JL2DMLLPOB5GHSPET544HAA, seg_pos={0 0}, upload=220713]: Uploading piece... Dec 16 03:12:18.766432894 (251h43m18.637515929s) [noise=true , address=77.90.31.16:4735 , piece_id=ATZ3AJVXFRAXDM4DR6MKDEHTRHM6AG4RB4YLKDHDLDMO5XFF3MPA, seg_pos={0 0}, upload=220713]: Uploading piece... Dec 16 03:12:18.766736152 (251h43m18.637819187s) [noise=true , address=145.79.227.52:4447 , piece_id=OYGRYLI43OEMB7MBUTGLXG76U4WUUR3EMNCUZQ5C67DVHLWBVHJA, seg_pos={0 0}, upload=220713]: Uploading piece... Dec 16 03:12:20.470095008 (251h43m20.341178064s) [upload=220713]: Next returned nil segment Dec 16 03:12:20.470146787 (251h43m20.341229832s) [upload=220713]: Waiting for error group managing segments... Dec 16 03:12:20.753665772 (251h43m20.624748807s) [noise=true , address=145.79.227.42:4123 , piece_id=NMJHDWWFESF3GM5TMEZFQGFD7YY5QHIIYQVHF3JIINSJHSEQDFZQ, seg_pos={0 0}, upload=220713]: Done uploading piece. err: <nil> Dec 16 03:12:20.757729490 (251h43m20.628812535s) [noise=true , address=145.79.227.29:4942 , piece_id=SON2DHJWJHPLZ7IJG6HFLK6D6C5O6JL2DMLLPOB5GHSPET544HAA, seg_pos={0 0}, upload=220713]: Done uploading piece. err: <nil> Dec 16 03:12:20.760602501 (251h43m20.631685546s) [noise=true , address=145.79.227.36:4518 , piece_id=I5C4LFPXUEB2JBUTWHXQV7VRJCCNN2RP7Z3CQ7RQENNMUBNNMHYQ, seg_pos={0 0}, upload=220713]: Done uploading piece. err: <nil> Dec 16 03:12:20.764326061 (251h43m20.635409106s) [noise=true , address=145.79.227.25:4617 , piece_id=TCLE2SK75NCWPLGUPYLJI3M5WD75Z2APQUGUXYJFFYAH5MU42T6A, seg_pos={0 0}, upload=220713]: Done uploading piece. err: <nil> Dec 16 03:12:20.766406653 (251h43m20.637489698s) [noise=true , address=145.79.227.37:4565 , piece_id=4VFLWAOYZ7OOHBEDWSBLAWDZODVQEU3MFCQKSNGUE6HLC3MDPVCQ, seg_pos={0 0}, upload=220713]: Done uploading piece. err: <nil> Dec 16 03:12:20.770108361 (251h43m20.641191406s) [noise=true , address=145.79.227.33:4870 , piece_id=MRUDYCE63CU2SHIFP64BNHFRLYAO2A3KXN6GIZGG7QJGX6S7UXDA, seg_pos={0 0}, upload=220713]: Done uploading piece. err: <nil> Dec 16 03:12:20.773437491 (251h43m20.644520546s) [noise=true , address=145.79.227.50:4373 , piece_id=YMG7Q5VQQKAYHO2HAEPOBRRDARQ25HGALNK3YOMIXUXYQONNKXSA, seg_pos={0 0}, upload=220713]: Done uploading piece. err: <nil> Dec 16 03:12:20.775082453 (251h43m20.646165488s) [noise=true , address=145.79.227.52:4447 , piece_id=OYGRYLI43OEMB7MBUTGLXG76U4WUUR3EMNCUZQ5C67DVHLWBVHJA, seg_pos={0 0}, upload=220713]: Done uploading piece. err: <nil> Dec 16 03:12:20.833660625 (251h43m20.70474366s) [noise=true , address=145.79.227.51:4409 , piece_id=3IEX2COS67O2QHKMMPC2NZJ3PGDRDG3O5WOKIY5IA4HKMZFDNF4Q, seg_pos={0 0}, upload=220713]: Done uploading piece. err: <nil> Dec 16 03:12:20.844657545 (251h43m20.71574058s) [noise=true , address=145.79.227.14:4840 , piece_id=3ABV6MR5KUUYYF7I2TU5MEXF5ULEY6IU6YIF72DCGR5MBALRWF3A, seg_pos={0 0}, upload=220713]: Done uploading piece. err: <nil> Dec 16 03:12:20.859069218 (251h43m20.730152273s) [noise=true , address=145.79.227.32:4834 , piece_id=RRMTYRQ37P6YWUV6BPVUC2BFLCRPWN6Q7VEVX6BDA54IKWTFH7SQ, seg_pos={0 0}, upload=220713]: Done uploading piece. err: <nil> Dec 16 03:12:21.007293480 (251h43m20.878376515s) [noise=true , address=77.90.31.16:4735 , piece_id=ATZ3AJVXFRAXDM4DR6MKDEHTRHM6AG4RB4YLKDHDLDMO5XFF3MPA, seg_pos={0 0}, upload=220713]: Done uploading piece. err: <nil> Dec 16 03:12:21.014381907 (251h43m20.885464952s) [noise=true , address=45.91.116.5:4298 , piece_id=GVZCYRIPTXYE6WOPQPRYKJSGV23I3GZ22SOCOJSXFUBLNDVQN2MA, seg_pos={0 0}, upload=220713]: Done uploading piece. err: <nil> Dec 16 03:12:21.014403138 (251h43m20.885486173s) [seg_pos={0 0}, upload=220713]: Segment reached optimal threshold of 13 pieces. Dec 16 03:12:21.014711706 (251h43m20.885794741s) [noise=true , address=145.79.227.44:4194 , piece_id=RJPVGEHG2XIR5JAPDFN6FR5ZH7ICFU2ZQIQKURWSLTBQT4UV7POA, seg_pos={0 0}, upload=220713]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh): protocol: expected piece hash; context canceled; EOF Dec 16 03:12:21.014721354 (251h43m20.885804389s) [noise=true , address=145.79.227.44:4194 , piece_id=RJPVGEHG2XIR5JAPDFN6FR5ZH7ICFU2ZQIQKURWSLTBQT4UV7POA, seg_pos={0 0}, upload=220713]: Long tail cancellation due to optimal threshold Dec 16 03:12:21.014764125 (251h43m20.88584716s) [noise=true , address=45.91.116.6:4974 , piece_id=GQDHG4LFMNUHAKFA3GA3WANCW5OGHOWCYM3IZCUGTWI54LGFWDDA, seg_pos={0 0}, upload=220713]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: expected piece hash; context canceled; EOF Dec 16 03:12:21.014782741 (251h43m20.885865786s) [noise=true , address=45.91.116.6:4974 , piece_id=GQDHG4LFMNUHAKFA3GA3WANCW5OGHOWCYM3IZCUGTWI54LGFWDDA, seg_pos={0 0}, upload=220713]: Long tail cancellation due to optimal threshold Dec 16 03:12:21.014804422 (251h43m20.885887467s) [noise=true , address=77.90.31.10:4520 , piece_id=Q55PSEFHSAOGY7GD4DWI2G56U66BXVZHXZ52DX4LYV4B52F5ZZ6A, seg_pos={0 0}, upload=220713]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp): protocol: expected piece hash; context canceled; EOF Dec 16 03:12:21.014809482 (251h43m20.885892527s) [noise=true , address=77.90.31.10:4520 , piece_id=Q55PSEFHSAOGY7GD4DWI2G56U66BXVZHXZ52DX4LYV4B52F5ZZ6A, seg_pos={0 0}, upload=220713]: Long tail cancellation due to optimal threshold Dec 16 03:12:21.014826394 (251h43m20.885909429s) [seg_pos={0 0}, upload=220713]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 03:12:21.015234772 (251h43m20.886317807s) [upload=220713]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 03:12:21.015241204 (251h43m20.886324249s) [upload=220713]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 03:12:21.319403538 (251h43m21.190486583s) [upload=220713]: Done uploading segments... Dec 16 03:12:21.319427222 (251h43m21.190510257s) [upload=220713]: Upload finished. err: <nil> Duration: 5.978735228s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 6.085945303s trace: https://jaeger.dev.storj.io/trace/2e0c5260bf6208b3 FAIL (2025-12-16 03:15:13 UTC) orbiter warning: context deadline exceeded Duration: 6.085977314s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 5.902767471s trace: https://jaeger.dev.storj.io/trace/13f785634570d9fc FAIL (2025-12-16 03:15:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 16 03:12:18.507971969 (251h43m18.379055014s) [upload=220713]: Starting upload Dec 16 03:12:18.508025069 (251h43m18.379108115s) [upload=220713]: Done starting upload Dec 16 03:12:18.508197317 (251h43m18.379280353s) [upload=220713]: Uploading segments... Dec 16 03:12:18.508315022 (251h43m18.379398067s) [upload=220713]: Got next segment. Inline: false Dec 16 03:12:18.509349181 (251h43m18.380432216s) [upload=220713]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 03:12:18.509360643 (251h43m18.380443688s) [upload=220713]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 03:12:18.761879946 (251h43m18.632962991s) [seg_pos={0 0}, upload=220713]: Begin upload segment... Dec 16 03:12:18.761929029 (251h43m18.633012064s) [seg_pos={0 0}, upload=220713]: Done begin upload segment. Dec 16 03:12:18.761967552 (251h43m18.633050597s) [noise=true , address=77.90.31.10:4520 , piece_id=Q55PSEFHSAOGY7GD4DWI2G56U66BXVZHXZ52DX4LYV4B52F5ZZ6A, seg_pos={0 0}, upload=220713]: Uploading piece... Dec 16 03:12:18.762560491 (251h43m18.633643536s) [noise=true , address=45.91.116.6:4974 , piece_id=GQDHG4LFMNUHAKFA3GA3WANCW5OGHOWCYM3IZCUGTWI54LGFWDDA, seg_pos={0 0}, upload=220713]: Uploading piece... Dec 16 03:12:18.762660291 (251h43m18.633743336s) [noise=true , address=145.79.227.25:4617 , piece_id=TCLE2SK75NCWPLGUPYLJI3M5WD75Z2APQUGUXYJFFYAH5MU42T6A, seg_pos={0 0}, upload=220713]: Uploading piece... Dec 16 03:12:18.763036347 (251h43m18.634119382s) [noise=true , address=145.79.227.36:4518 , piece_id=I5C4LFPXUEB2JBUTWHXQV7VRJCCNN2RP7Z3CQ7RQENNMUBNNMHYQ, seg_pos={0 0}, upload=220713]: Uploading piece... Dec 16 03:12:18.763318324 (251h43m18.634401359s) [noise=true , address=145.79.227.14:4840 , piece_id=3ABV6MR5KUUYYF7I2TU5MEXF5ULEY6IU6YIF72DCGR5MBALRWF3A, seg_pos={0 0}, upload=220713]: Uploading piece... Dec 16 03:12:18.763607314 (251h43m18.634690349s) [noise=true , address=145.79.227.50:4373 , piece_id=YMG7Q5VQQKAYHO2HAEPOBRRDARQ25HGALNK3YOMIXUXYQONNKXSA, seg_pos={0 0}, upload=220713]: Uploading piece... Dec 16 03:12:18.763864645 (251h43m18.63494769s) [noise=true , address=145.79.227.51:4409 , piece_id=3IEX2COS67O2QHKMMPC2NZJ3PGDRDG3O5WOKIY5IA4HKMZFDNF4Q, seg_pos={0 0}, upload=220713]: Uploading piece... Dec 16 03:12:18.763888079 (251h43m18.634971154s) [noise=true , address=145.79.227.42:4123 , piece_id=NMJHDWWFESF3GM5TMEZFQGFD7YY5QHIIYQVHF3JIINSJHSEQDFZQ, seg_pos={0 0}, upload=220713]: Uploading piece... Dec 16 03:12:18.763947542 (251h43m18.635030607s) [noise=true , address=145.79.227.32:4834 , piece_id=RRMTYRQ37P6YWUV6BPVUC2BFLCRPWN6Q7VEVX6BDA54IKWTFH7SQ, seg_pos={0 0}, upload=220713]: Uploading piece... Dec 16 03:12:18.764263093 (251h43m18.635346138s) [noise=true , address=145.79.227.37:4565 , piece_id=4VFLWAOYZ7OOHBEDWSBLAWDZODVQEU3MFCQKSNGUE6HLC3MDPVCQ, seg_pos={0 0}, upload=220713]: Uploading piece... Dec 16 03:12:18.764390716 (251h43m18.635473761s) [noise=true , address=145.79.227.44:4194 , piece_id=RJPVGEHG2XIR5JAPDFN6FR5ZH7ICFU2ZQIQKURWSLTBQT4UV7POA, seg_pos={0 0}, upload=220713]: Uploading piece... Dec 16 03:12:18.764941455 (251h43m18.6360245s) [noise=true , address=145.79.227.33:4870 , piece_id=MRUDYCE63CU2SHIFP64BNHFRLYAO2A3KXN6GIZGG7QJGX6S7UXDA, seg_pos={0 0}, upload=220713]: Uploading piece... Dec 16 03:12:18.765504287 (251h43m18.636587322s) [noise=true , address=45.91.116.5:4298 , piece_id=GVZCYRIPTXYE6WOPQPRYKJSGV23I3GZ22SOCOJSXFUBLNDVQN2MA, seg_pos={0 0}, upload=220713]: Uploading piece... Dec 16 03:12:18.766127143 (251h43m18.637210188s) [noise=true , address=145.79.227.29:4942 , piece_id=SON2DHJWJHPLZ7IJG6HFLK6D6C5O6JL2DMLLPOB5GHSPET544HAA, seg_pos={0 0}, upload=220713]: Uploading piece... Dec 16 03:12:18.766432894 (251h43m18.637515929s) [noise=true , address=77.90.31.16:4735 , piece_id=ATZ3AJVXFRAXDM4DR6MKDEHTRHM6AG4RB4YLKDHDLDMO5XFF3MPA, seg_pos={0 0}, upload=220713]: Uploading piece... Dec 16 03:12:18.766736152 (251h43m18.637819187s) [noise=true , address=145.79.227.52:4447 , piece_id=OYGRYLI43OEMB7MBUTGLXG76U4WUUR3EMNCUZQ5C67DVHLWBVHJA, seg_pos={0 0}, upload=220713]: Uploading piece... Dec 16 03:12:20.470095008 (251h43m20.341178064s) [upload=220713]: Next returned nil segment Dec 16 03:12:20.470146787 (251h43m20.341229832s) [upload=220713]: Waiting for error group managing segments... Dec 16 03:12:20.753665772 (251h43m20.624748807s) [noise=true , address=145.79.227.42:4123 , piece_id=NMJHDWWFESF3GM5TMEZFQGFD7YY5QHIIYQVHF3JIINSJHSEQDFZQ, seg_pos={0 0}, upload=220713]: Done uploading piece. err: <nil> Dec 16 03:12:20.757729490 (251h43m20.628812535s) [noise=true , address=145.79.227.29:4942 , piece_id=SON2DHJWJHPLZ7IJG6HFLK6D6C5O6JL2DMLLPOB5GHSPET544HAA, seg_pos={0 0}, upload=220713]: Done uploading piece. err: <nil> Dec 16 03:12:20.760602501 (251h43m20.631685546s) [noise=true , address=145.79.227.36:4518 , piece_id=I5C4LFPXUEB2JBUTWHXQV7VRJCCNN2RP7Z3CQ7RQENNMUBNNMHYQ, seg_pos={0 0}, upload=220713]: Done uploading piece. err: <nil> Dec 16 03:12:20.764326061 (251h43m20.635409106s) [noise=true , address=145.79.227.25:4617 , piece_id=TCLE2SK75NCWPLGUPYLJI3M5WD75Z2APQUGUXYJFFYAH5MU42T6A, seg_pos={0 0}, upload=220713]: Done uploading piece. err: <nil> Dec 16 03:12:20.766406653 (251h43m20.637489698s) [noise=true , address=145.79.227.37:4565 , piece_id=4VFLWAOYZ7OOHBEDWSBLAWDZODVQEU3MFCQKSNGUE6HLC3MDPVCQ, seg_pos={0 0}, upload=220713]: Done uploading piece. err: <nil> Dec 16 03:12:20.770108361 (251h43m20.641191406s) [noise=true , address=145.79.227.33:4870 , piece_id=MRUDYCE63CU2SHIFP64BNHFRLYAO2A3KXN6GIZGG7QJGX6S7UXDA, seg_pos={0 0}, upload=220713]: Done uploading piece. err: <nil> Dec 16 03:12:20.773437491 (251h43m20.644520546s) [noise=true , address=145.79.227.50:4373 , piece_id=YMG7Q5VQQKAYHO2HAEPOBRRDARQ25HGALNK3YOMIXUXYQONNKXSA, seg_pos={0 0}, upload=220713]: Done uploading piece. err: <nil> Dec 16 03:12:20.775082453 (251h43m20.646165488s) [noise=true , address=145.79.227.52:4447 , piece_id=OYGRYLI43OEMB7MBUTGLXG76U4WUUR3EMNCUZQ5C67DVHLWBVHJA, seg_pos={0 0}, upload=220713]: Done uploading piece. err: <nil> Dec 16 03:12:20.833660625 (251h43m20.70474366s) [noise=true , address=145.79.227.51:4409 , piece_id=3IEX2COS67O2QHKMMPC2NZJ3PGDRDG3O5WOKIY5IA4HKMZFDNF4Q, seg_pos={0 0}, upload=220713]: Done uploading piece. err: <nil> Dec 16 03:12:20.844657545 (251h43m20.71574058s) [noise=true , address=145.79.227.14:4840 , piece_id=3ABV6MR5KUUYYF7I2TU5MEXF5ULEY6IU6YIF72DCGR5MBALRWF3A, seg_pos={0 0}, upload=220713]: Done uploading piece. err: <nil> Dec 16 03:12:20.859069218 (251h43m20.730152273s) [noise=true , address=145.79.227.32:4834 , piece_id=RRMTYRQ37P6YWUV6BPVUC2BFLCRPWN6Q7VEVX6BDA54IKWTFH7SQ, seg_pos={0 0}, upload=220713]: Done uploading piece. err: <nil> Dec 16 03:12:21.007293480 (251h43m20.878376515s) [noise=true , address=77.90.31.16:4735 , piece_id=ATZ3AJVXFRAXDM4DR6MKDEHTRHM6AG4RB4YLKDHDLDMO5XFF3MPA, seg_pos={0 0}, upload=220713]: Done uploading piece. err: <nil> Dec 16 03:12:21.014381907 (251h43m20.885464952s) [noise=true , address=45.91.116.5:4298 , piece_id=GVZCYRIPTXYE6WOPQPRYKJSGV23I3GZ22SOCOJSXFUBLNDVQN2MA, seg_pos={0 0}, upload=220713]: Done uploading piece. err: <nil> Dec 16 03:12:21.014403138 (251h43m20.885486173s) [seg_pos={0 0}, upload=220713]: Segment reached optimal threshold of 13 pieces. Dec 16 03:12:21.014711706 (251h43m20.885794741s) [noise=true , address=145.79.227.44:4194 , piece_id=RJPVGEHG2XIR5JAPDFN6FR5ZH7ICFU2ZQIQKURWSLTBQT4UV7POA, seg_pos={0 0}, upload=220713]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh): protocol: expected piece hash; context canceled; EOF Dec 16 03:12:21.014721354 (251h43m20.885804389s) [noise=true , address=145.79.227.44:4194 , piece_id=RJPVGEHG2XIR5JAPDFN6FR5ZH7ICFU2ZQIQKURWSLTBQT4UV7POA, seg_pos={0 0}, upload=220713]: Long tail cancellation due to optimal threshold Dec 16 03:12:21.014764125 (251h43m20.88584716s) [noise=true , address=45.91.116.6:4974 , piece_id=GQDHG4LFMNUHAKFA3GA3WANCW5OGHOWCYM3IZCUGTWI54LGFWDDA, seg_pos={0 0}, upload=220713]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: expected piece hash; context canceled; EOF Dec 16 03:12:21.014782741 (251h43m20.885865786s) [noise=true , address=45.91.116.6:4974 , piece_id=GQDHG4LFMNUHAKFA3GA3WANCW5OGHOWCYM3IZCUGTWI54LGFWDDA, seg_pos={0 0}, upload=220713]: Long tail cancellation due to optimal threshold Dec 16 03:12:21.014804422 (251h43m20.885887467s) [noise=true , address=77.90.31.10:4520 , piece_id=Q55PSEFHSAOGY7GD4DWI2G56U66BXVZHXZ52DX4LYV4B52F5ZZ6A, seg_pos={0 0}, upload=220713]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp): protocol: expected piece hash; context canceled; EOF Dec 16 03:12:21.014809482 (251h43m20.885892527s) [noise=true , address=77.90.31.10:4520 , piece_id=Q55PSEFHSAOGY7GD4DWI2G56U66BXVZHXZ52DX4LYV4B52F5ZZ6A, seg_pos={0 0}, upload=220713]: Long tail cancellation due to optimal threshold Dec 16 03:12:21.014826394 (251h43m20.885909429s) [seg_pos={0 0}, upload=220713]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 03:12:21.015234772 (251h43m20.886317807s) [upload=220713]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 03:12:21.015241204 (251h43m20.886324249s) [upload=220713]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 03:12:21.319403538 (251h43m21.190486583s) [upload=220713]: Done uploading segments... Dec 16 03:12:21.319427222 (251h43m21.190510257s) [upload=220713]: Upload finished. err: <nil> Duration: 5.902796205s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 6.099094461s trace: https://jaeger.dev.storj.io/trace/2e0c5260bf6208b3 FAIL (2025-12-16 03:14:57 UTC) orbiter warning: context deadline exceeded Duration: 6.099252001s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 5.788862868s trace: https://jaeger.dev.storj.io/trace/13f785634570d9fc FAIL (2025-12-16 03:14:26 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 16 03:12:18.507971969 (251h43m18.379055014s) [upload=220713]: Starting upload Dec 16 03:12:18.508025069 (251h43m18.379108115s) [upload=220713]: Done starting upload Dec 16 03:12:18.508197317 (251h43m18.379280353s) [upload=220713]: Uploading segments... Dec 16 03:12:18.508315022 (251h43m18.379398067s) [upload=220713]: Got next segment. Inline: false Dec 16 03:12:18.509349181 (251h43m18.380432216s) [upload=220713]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 03:12:18.509360643 (251h43m18.380443688s) [upload=220713]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 03:12:18.761879946 (251h43m18.632962991s) [seg_pos={0 0}, upload=220713]: Begin upload segment... Dec 16 03:12:18.761929029 (251h43m18.633012064s) [seg_pos={0 0}, upload=220713]: Done begin upload segment. Dec 16 03:12:18.761967552 (251h43m18.633050597s) [noise=true , address=77.90.31.10:4520 , piece_id=Q55PSEFHSAOGY7GD4DWI2G56U66BXVZHXZ52DX4LYV4B52F5ZZ6A, seg_pos={0 0}, upload=220713]: Uploading piece... Dec 16 03:12:18.762560491 (251h43m18.633643536s) [noise=true , address=45.91.116.6:4974 , piece_id=GQDHG4LFMNUHAKFA3GA3WANCW5OGHOWCYM3IZCUGTWI54LGFWDDA, seg_pos={0 0}, upload=220713]: Uploading piece... Dec 16 03:12:18.762660291 (251h43m18.633743336s) [noise=true , address=145.79.227.25:4617 , piece_id=TCLE2SK75NCWPLGUPYLJI3M5WD75Z2APQUGUXYJFFYAH5MU42T6A, seg_pos={0 0}, upload=220713]: Uploading piece... Dec 16 03:12:18.763036347 (251h43m18.634119382s) [noise=true , address=145.79.227.36:4518 , piece_id=I5C4LFPXUEB2JBUTWHXQV7VRJCCNN2RP7Z3CQ7RQENNMUBNNMHYQ, seg_pos={0 0}, upload=220713]: Uploading piece... Dec 16 03:12:18.763318324 (251h43m18.634401359s) [noise=true , address=145.79.227.14:4840 , piece_id=3ABV6MR5KUUYYF7I2TU5MEXF5ULEY6IU6YIF72DCGR5MBALRWF3A, seg_pos={0 0}, upload=220713]: Uploading piece... Dec 16 03:12:18.763607314 (251h43m18.634690349s) [noise=true , address=145.79.227.50:4373 , piece_id=YMG7Q5VQQKAYHO2HAEPOBRRDARQ25HGALNK3YOMIXUXYQONNKXSA, seg_pos={0 0}, upload=220713]: Uploading piece... Dec 16 03:12:18.763864645 (251h43m18.63494769s) [noise=true , address=145.79.227.51:4409 , piece_id=3IEX2COS67O2QHKMMPC2NZJ3PGDRDG3O5WOKIY5IA4HKMZFDNF4Q, seg_pos={0 0}, upload=220713]: Uploading piece... Dec 16 03:12:18.763888079 (251h43m18.634971154s) [noise=true , address=145.79.227.42:4123 , piece_id=NMJHDWWFESF3GM5TMEZFQGFD7YY5QHIIYQVHF3JIINSJHSEQDFZQ, seg_pos={0 0}, upload=220713]: Uploading piece... Dec 16 03:12:18.763947542 (251h43m18.635030607s) [noise=true , address=145.79.227.32:4834 , piece_id=RRMTYRQ37P6YWUV6BPVUC2BFLCRPWN6Q7VEVX6BDA54IKWTFH7SQ, seg_pos={0 0}, upload=220713]: Uploading piece... Dec 16 03:12:18.764263093 (251h43m18.635346138s) [noise=true , address=145.79.227.37:4565 , piece_id=4VFLWAOYZ7OOHBEDWSBLAWDZODVQEU3MFCQKSNGUE6HLC3MDPVCQ, seg_pos={0 0}, upload=220713]: Uploading piece... Dec 16 03:12:18.764390716 (251h43m18.635473761s) [noise=true , address=145.79.227.44:4194 , piece_id=RJPVGEHG2XIR5JAPDFN6FR5ZH7ICFU2ZQIQKURWSLTBQT4UV7POA, seg_pos={0 0}, upload=220713]: Uploading piece... Dec 16 03:12:18.764941455 (251h43m18.6360245s) [noise=true , address=145.79.227.33:4870 , piece_id=MRUDYCE63CU2SHIFP64BNHFRLYAO2A3KXN6GIZGG7QJGX6S7UXDA, seg_pos={0 0}, upload=220713]: Uploading piece... Dec 16 03:12:18.765504287 (251h43m18.636587322s) [noise=true , address=45.91.116.5:4298 , piece_id=GVZCYRIPTXYE6WOPQPRYKJSGV23I3GZ22SOCOJSXFUBLNDVQN2MA, seg_pos={0 0}, upload=220713]: Uploading piece... Dec 16 03:12:18.766127143 (251h43m18.637210188s) [noise=true , address=145.79.227.29:4942 , piece_id=SON2DHJWJHPLZ7IJG6HFLK6D6C5O6JL2DMLLPOB5GHSPET544HAA, seg_pos={0 0}, upload=220713]: Uploading piece... Dec 16 03:12:18.766432894 (251h43m18.637515929s) [noise=true , address=77.90.31.16:4735 , piece_id=ATZ3AJVXFRAXDM4DR6MKDEHTRHM6AG4RB4YLKDHDLDMO5XFF3MPA, seg_pos={0 0}, upload=220713]: Uploading piece... Dec 16 03:12:18.766736152 (251h43m18.637819187s) [noise=true , address=145.79.227.52:4447 , piece_id=OYGRYLI43OEMB7MBUTGLXG76U4WUUR3EMNCUZQ5C67DVHLWBVHJA, seg_pos={0 0}, upload=220713]: Uploading piece... Dec 16 03:12:20.470095008 (251h43m20.341178064s) [upload=220713]: Next returned nil segment Dec 16 03:12:20.470146787 (251h43m20.341229832s) [upload=220713]: Waiting for error group managing segments... Dec 16 03:12:20.753665772 (251h43m20.624748807s) [noise=true , address=145.79.227.42:4123 , piece_id=NMJHDWWFESF3GM5TMEZFQGFD7YY5QHIIYQVHF3JIINSJHSEQDFZQ, seg_pos={0 0}, upload=220713]: Done uploading piece. err: <nil> Dec 16 03:12:20.757729490 (251h43m20.628812535s) [noise=true , address=145.79.227.29:4942 , piece_id=SON2DHJWJHPLZ7IJG6HFLK6D6C5O6JL2DMLLPOB5GHSPET544HAA, seg_pos={0 0}, upload=220713]: Done uploading piece. err: <nil> Dec 16 03:12:20.760602501 (251h43m20.631685546s) [noise=true , address=145.79.227.36:4518 , piece_id=I5C4LFPXUEB2JBUTWHXQV7VRJCCNN2RP7Z3CQ7RQENNMUBNNMHYQ, seg_pos={0 0}, upload=220713]: Done uploading piece. err: <nil> Dec 16 03:12:20.764326061 (251h43m20.635409106s) [noise=true , address=145.79.227.25:4617 , piece_id=TCLE2SK75NCWPLGUPYLJI3M5WD75Z2APQUGUXYJFFYAH5MU42T6A, seg_pos={0 0}, upload=220713]: Done uploading piece. err: <nil> Dec 16 03:12:20.766406653 (251h43m20.637489698s) [noise=true , address=145.79.227.37:4565 , piece_id=4VFLWAOYZ7OOHBEDWSBLAWDZODVQEU3MFCQKSNGUE6HLC3MDPVCQ, seg_pos={0 0}, upload=220713]: Done uploading piece. err: <nil> Dec 16 03:12:20.770108361 (251h43m20.641191406s) [noise=true , address=145.79.227.33:4870 , piece_id=MRUDYCE63CU2SHIFP64BNHFRLYAO2A3KXN6GIZGG7QJGX6S7UXDA, seg_pos={0 0}, upload=220713]: Done uploading piece. err: <nil> Dec 16 03:12:20.773437491 (251h43m20.644520546s) [noise=true , address=145.79.227.50:4373 , piece_id=YMG7Q5VQQKAYHO2HAEPOBRRDARQ25HGALNK3YOMIXUXYQONNKXSA, seg_pos={0 0}, upload=220713]: Done uploading piece. err: <nil> Dec 16 03:12:20.775082453 (251h43m20.646165488s) [noise=true , address=145.79.227.52:4447 , piece_id=OYGRYLI43OEMB7MBUTGLXG76U4WUUR3EMNCUZQ5C67DVHLWBVHJA, seg_pos={0 0}, upload=220713]: Done uploading piece. err: <nil> Dec 16 03:12:20.833660625 (251h43m20.70474366s) [noise=true , address=145.79.227.51:4409 , piece_id=3IEX2COS67O2QHKMMPC2NZJ3PGDRDG3O5WOKIY5IA4HKMZFDNF4Q, seg_pos={0 0}, upload=220713]: Done uploading piece. err: <nil> Dec 16 03:12:20.844657545 (251h43m20.71574058s) [noise=true , address=145.79.227.14:4840 , piece_id=3ABV6MR5KUUYYF7I2TU5MEXF5ULEY6IU6YIF72DCGR5MBALRWF3A, seg_pos={0 0}, upload=220713]: Done uploading piece. err: <nil> Dec 16 03:12:20.859069218 (251h43m20.730152273s) [noise=true , address=145.79.227.32:4834 , piece_id=RRMTYRQ37P6YWUV6BPVUC2BFLCRPWN6Q7VEVX6BDA54IKWTFH7SQ, seg_pos={0 0}, upload=220713]: Done uploading piece. err: <nil> Dec 16 03:12:21.007293480 (251h43m20.878376515s) [noise=true , address=77.90.31.16:4735 , piece_id=ATZ3AJVXFRAXDM4DR6MKDEHTRHM6AG4RB4YLKDHDLDMO5XFF3MPA, seg_pos={0 0}, upload=220713]: Done uploading piece. err: <nil> Dec 16 03:12:21.014381907 (251h43m20.885464952s) [noise=true , address=45.91.116.5:4298 , piece_id=GVZCYRIPTXYE6WOPQPRYKJSGV23I3GZ22SOCOJSXFUBLNDVQN2MA, seg_pos={0 0}, upload=220713]: Done uploading piece. err: <nil> Dec 16 03:12:21.014403138 (251h43m20.885486173s) [seg_pos={0 0}, upload=220713]: Segment reached optimal threshold of 13 pieces. Dec 16 03:12:21.014711706 (251h43m20.885794741s) [noise=true , address=145.79.227.44:4194 , piece_id=RJPVGEHG2XIR5JAPDFN6FR5ZH7ICFU2ZQIQKURWSLTBQT4UV7POA, seg_pos={0 0}, upload=220713]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh): protocol: expected piece hash; context canceled; EOF Dec 16 03:12:21.014721354 (251h43m20.885804389s) [noise=true , address=145.79.227.44:4194 , piece_id=RJPVGEHG2XIR5JAPDFN6FR5ZH7ICFU2ZQIQKURWSLTBQT4UV7POA, seg_pos={0 0}, upload=220713]: Long tail cancellation due to optimal threshold Dec 16 03:12:21.014764125 (251h43m20.88584716s) [noise=true , address=45.91.116.6:4974 , piece_id=GQDHG4LFMNUHAKFA3GA3WANCW5OGHOWCYM3IZCUGTWI54LGFWDDA, seg_pos={0 0}, upload=220713]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: expected piece hash; context canceled; EOF Dec 16 03:12:21.014782741 (251h43m20.885865786s) [noise=true , address=45.91.116.6:4974 , piece_id=GQDHG4LFMNUHAKFA3GA3WANCW5OGHOWCYM3IZCUGTWI54LGFWDDA, seg_pos={0 0}, upload=220713]: Long tail cancellation due to optimal threshold Dec 16 03:12:21.014804422 (251h43m20.885887467s) [noise=true , address=77.90.31.10:4520 , piece_id=Q55PSEFHSAOGY7GD4DWI2G56U66BXVZHXZ52DX4LYV4B52F5ZZ6A, seg_pos={0 0}, upload=220713]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp): protocol: expected piece hash; context canceled; EOF Dec 16 03:12:21.014809482 (251h43m20.885892527s) [noise=true , address=77.90.31.10:4520 , piece_id=Q55PSEFHSAOGY7GD4DWI2G56U66BXVZHXZ52DX4LYV4B52F5ZZ6A, seg_pos={0 0}, upload=220713]: Long tail cancellation due to optimal threshold Dec 16 03:12:21.014826394 (251h43m20.885909429s) [seg_pos={0 0}, upload=220713]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 03:12:21.015234772 (251h43m20.886317807s) [upload=220713]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 03:12:21.015241204 (251h43m20.886324249s) [upload=220713]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 03:12:21.319403538 (251h43m21.190486583s) [upload=220713]: Done uploading segments... Dec 16 03:12:21.319427222 (251h43m21.190510257s) [upload=220713]: Upload finished. err: <nil> Duration: 5.788887856s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 5.91258667s trace: https://jaeger.dev.storj.io/trace/2e0c5260bf6208b3 FAIL (2025-12-16 03:13:57 UTC) orbiter warning: context deadline exceeded Duration: 5.912868165s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 6.000490629s trace: https://jaeger.dev.storj.io/trace/13f785634570d9fc FAIL (2025-12-16 03:13:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 16 03:12:18.507971969 (251h43m18.379055014s) [upload=220713]: Starting upload Dec 16 03:12:18.508025069 (251h43m18.379108115s) [upload=220713]: Done starting upload Dec 16 03:12:18.508197317 (251h43m18.379280353s) [upload=220713]: Uploading segments... Dec 16 03:12:18.508315022 (251h43m18.379398067s) [upload=220713]: Got next segment. Inline: false Dec 16 03:12:18.509349181 (251h43m18.380432216s) [upload=220713]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 03:12:18.509360643 (251h43m18.380443688s) [upload=220713]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 03:12:18.761879946 (251h43m18.632962991s) [seg_pos={0 0}, upload=220713]: Begin upload segment... Dec 16 03:12:18.761929029 (251h43m18.633012064s) [seg_pos={0 0}, upload=220713]: Done begin upload segment. Dec 16 03:12:18.761967552 (251h43m18.633050597s) [noise=true , address=77.90.31.10:4520 , piece_id=Q55PSEFHSAOGY7GD4DWI2G56U66BXVZHXZ52DX4LYV4B52F5ZZ6A, seg_pos={0 0}, upload=220713]: Uploading piece... Dec 16 03:12:18.762560491 (251h43m18.633643536s) [noise=true , address=45.91.116.6:4974 , piece_id=GQDHG4LFMNUHAKFA3GA3WANCW5OGHOWCYM3IZCUGTWI54LGFWDDA, seg_pos={0 0}, upload=220713]: Uploading piece... Dec 16 03:12:18.762660291 (251h43m18.633743336s) [noise=true , address=145.79.227.25:4617 , piece_id=TCLE2SK75NCWPLGUPYLJI3M5WD75Z2APQUGUXYJFFYAH5MU42T6A, seg_pos={0 0}, upload=220713]: Uploading piece... Dec 16 03:12:18.763036347 (251h43m18.634119382s) [noise=true , address=145.79.227.36:4518 , piece_id=I5C4LFPXUEB2JBUTWHXQV7VRJCCNN2RP7Z3CQ7RQENNMUBNNMHYQ, seg_pos={0 0}, upload=220713]: Uploading piece... Dec 16 03:12:18.763318324 (251h43m18.634401359s) [noise=true , address=145.79.227.14:4840 , piece_id=3ABV6MR5KUUYYF7I2TU5MEXF5ULEY6IU6YIF72DCGR5MBALRWF3A, seg_pos={0 0}, upload=220713]: Uploading piece... Dec 16 03:12:18.763607314 (251h43m18.634690349s) [noise=true , address=145.79.227.50:4373 , piece_id=YMG7Q5VQQKAYHO2HAEPOBRRDARQ25HGALNK3YOMIXUXYQONNKXSA, seg_pos={0 0}, upload=220713]: Uploading piece... Dec 16 03:12:18.763864645 (251h43m18.63494769s) [noise=true , address=145.79.227.51:4409 , piece_id=3IEX2COS67O2QHKMMPC2NZJ3PGDRDG3O5WOKIY5IA4HKMZFDNF4Q, seg_pos={0 0}, upload=220713]: Uploading piece... Dec 16 03:12:18.763888079 (251h43m18.634971154s) [noise=true , address=145.79.227.42:4123 , piece_id=NMJHDWWFESF3GM5TMEZFQGFD7YY5QHIIYQVHF3JIINSJHSEQDFZQ, seg_pos={0 0}, upload=220713]: Uploading piece... Dec 16 03:12:18.763947542 (251h43m18.635030607s) [noise=true , address=145.79.227.32:4834 , piece_id=RRMTYRQ37P6YWUV6BPVUC2BFLCRPWN6Q7VEVX6BDA54IKWTFH7SQ, seg_pos={0 0}, upload=220713]: Uploading piece... Dec 16 03:12:18.764263093 (251h43m18.635346138s) [noise=true , address=145.79.227.37:4565 , piece_id=4VFLWAOYZ7OOHBEDWSBLAWDZODVQEU3MFCQKSNGUE6HLC3MDPVCQ, seg_pos={0 0}, upload=220713]: Uploading piece... Dec 16 03:12:18.764390716 (251h43m18.635473761s) [noise=true , address=145.79.227.44:4194 , piece_id=RJPVGEHG2XIR5JAPDFN6FR5ZH7ICFU2ZQIQKURWSLTBQT4UV7POA, seg_pos={0 0}, upload=220713]: Uploading piece... Dec 16 03:12:18.764941455 (251h43m18.6360245s) [noise=true , address=145.79.227.33:4870 , piece_id=MRUDYCE63CU2SHIFP64BNHFRLYAO2A3KXN6GIZGG7QJGX6S7UXDA, seg_pos={0 0}, upload=220713]: Uploading piece... Dec 16 03:12:18.765504287 (251h43m18.636587322s) [noise=true , address=45.91.116.5:4298 , piece_id=GVZCYRIPTXYE6WOPQPRYKJSGV23I3GZ22SOCOJSXFUBLNDVQN2MA, seg_pos={0 0}, upload=220713]: Uploading piece... Dec 16 03:12:18.766127143 (251h43m18.637210188s) [noise=true , address=145.79.227.29:4942 , piece_id=SON2DHJWJHPLZ7IJG6HFLK6D6C5O6JL2DMLLPOB5GHSPET544HAA, seg_pos={0 0}, upload=220713]: Uploading piece... Dec 16 03:12:18.766432894 (251h43m18.637515929s) [noise=true , address=77.90.31.16:4735 , piece_id=ATZ3AJVXFRAXDM4DR6MKDEHTRHM6AG4RB4YLKDHDLDMO5XFF3MPA, seg_pos={0 0}, upload=220713]: Uploading piece... Dec 16 03:12:18.766736152 (251h43m18.637819187s) [noise=true , address=145.79.227.52:4447 , piece_id=OYGRYLI43OEMB7MBUTGLXG76U4WUUR3EMNCUZQ5C67DVHLWBVHJA, seg_pos={0 0}, upload=220713]: Uploading piece... Dec 16 03:12:20.470095008 (251h43m20.341178064s) [upload=220713]: Next returned nil segment Dec 16 03:12:20.470146787 (251h43m20.341229832s) [upload=220713]: Waiting for error group managing segments... Dec 16 03:12:20.753665772 (251h43m20.624748807s) [noise=true , address=145.79.227.42:4123 , piece_id=NMJHDWWFESF3GM5TMEZFQGFD7YY5QHIIYQVHF3JIINSJHSEQDFZQ, seg_pos={0 0}, upload=220713]: Done uploading piece. err: <nil> Dec 16 03:12:20.757729490 (251h43m20.628812535s) [noise=true , address=145.79.227.29:4942 , piece_id=SON2DHJWJHPLZ7IJG6HFLK6D6C5O6JL2DMLLPOB5GHSPET544HAA, seg_pos={0 0}, upload=220713]: Done uploading piece. err: <nil> Dec 16 03:12:20.760602501 (251h43m20.631685546s) [noise=true , address=145.79.227.36:4518 , piece_id=I5C4LFPXUEB2JBUTWHXQV7VRJCCNN2RP7Z3CQ7RQENNMUBNNMHYQ, seg_pos={0 0}, upload=220713]: Done uploading piece. err: <nil> Dec 16 03:12:20.764326061 (251h43m20.635409106s) [noise=true , address=145.79.227.25:4617 , piece_id=TCLE2SK75NCWPLGUPYLJI3M5WD75Z2APQUGUXYJFFYAH5MU42T6A, seg_pos={0 0}, upload=220713]: Done uploading piece. err: <nil> Dec 16 03:12:20.766406653 (251h43m20.637489698s) [noise=true , address=145.79.227.37:4565 , piece_id=4VFLWAOYZ7OOHBEDWSBLAWDZODVQEU3MFCQKSNGUE6HLC3MDPVCQ, seg_pos={0 0}, upload=220713]: Done uploading piece. err: <nil> Dec 16 03:12:20.770108361 (251h43m20.641191406s) [noise=true , address=145.79.227.33:4870 , piece_id=MRUDYCE63CU2SHIFP64BNHFRLYAO2A3KXN6GIZGG7QJGX6S7UXDA, seg_pos={0 0}, upload=220713]: Done uploading piece. err: <nil> Dec 16 03:12:20.773437491 (251h43m20.644520546s) [noise=true , address=145.79.227.50:4373 , piece_id=YMG7Q5VQQKAYHO2HAEPOBRRDARQ25HGALNK3YOMIXUXYQONNKXSA, seg_pos={0 0}, upload=220713]: Done uploading piece. err: <nil> Dec 16 03:12:20.775082453 (251h43m20.646165488s) [noise=true , address=145.79.227.52:4447 , piece_id=OYGRYLI43OEMB7MBUTGLXG76U4WUUR3EMNCUZQ5C67DVHLWBVHJA, seg_pos={0 0}, upload=220713]: Done uploading piece. err: <nil> Dec 16 03:12:20.833660625 (251h43m20.70474366s) [noise=true , address=145.79.227.51:4409 , piece_id=3IEX2COS67O2QHKMMPC2NZJ3PGDRDG3O5WOKIY5IA4HKMZFDNF4Q, seg_pos={0 0}, upload=220713]: Done uploading piece. err: <nil> Dec 16 03:12:20.844657545 (251h43m20.71574058s) [noise=true , address=145.79.227.14:4840 , piece_id=3ABV6MR5KUUYYF7I2TU5MEXF5ULEY6IU6YIF72DCGR5MBALRWF3A, seg_pos={0 0}, upload=220713]: Done uploading piece. err: <nil> Dec 16 03:12:20.859069218 (251h43m20.730152273s) [noise=true , address=145.79.227.32:4834 , piece_id=RRMTYRQ37P6YWUV6BPVUC2BFLCRPWN6Q7VEVX6BDA54IKWTFH7SQ, seg_pos={0 0}, upload=220713]: Done uploading piece. err: <nil> Dec 16 03:12:21.007293480 (251h43m20.878376515s) [noise=true , address=77.90.31.16:4735 , piece_id=ATZ3AJVXFRAXDM4DR6MKDEHTRHM6AG4RB4YLKDHDLDMO5XFF3MPA, seg_pos={0 0}, upload=220713]: Done uploading piece. err: <nil> Dec 16 03:12:21.014381907 (251h43m20.885464952s) [noise=true , address=45.91.116.5:4298 , piece_id=GVZCYRIPTXYE6WOPQPRYKJSGV23I3GZ22SOCOJSXFUBLNDVQN2MA, seg_pos={0 0}, upload=220713]: Done uploading piece. err: <nil> Dec 16 03:12:21.014403138 (251h43m20.885486173s) [seg_pos={0 0}, upload=220713]: Segment reached optimal threshold of 13 pieces. Dec 16 03:12:21.014711706 (251h43m20.885794741s) [noise=true , address=145.79.227.44:4194 , piece_id=RJPVGEHG2XIR5JAPDFN6FR5ZH7ICFU2ZQIQKURWSLTBQT4UV7POA, seg_pos={0 0}, upload=220713]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh): protocol: expected piece hash; context canceled; EOF Dec 16 03:12:21.014721354 (251h43m20.885804389s) [noise=true , address=145.79.227.44:4194 , piece_id=RJPVGEHG2XIR5JAPDFN6FR5ZH7ICFU2ZQIQKURWSLTBQT4UV7POA, seg_pos={0 0}, upload=220713]: Long tail cancellation due to optimal threshold Dec 16 03:12:21.014764125 (251h43m20.88584716s) [noise=true , address=45.91.116.6:4974 , piece_id=GQDHG4LFMNUHAKFA3GA3WANCW5OGHOWCYM3IZCUGTWI54LGFWDDA, seg_pos={0 0}, upload=220713]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: expected piece hash; context canceled; EOF Dec 16 03:12:21.014782741 (251h43m20.885865786s) [noise=true , address=45.91.116.6:4974 , piece_id=GQDHG4LFMNUHAKFA3GA3WANCW5OGHOWCYM3IZCUGTWI54LGFWDDA, seg_pos={0 0}, upload=220713]: Long tail cancellation due to optimal threshold Dec 16 03:12:21.014804422 (251h43m20.885887467s) [noise=true , address=77.90.31.10:4520 , piece_id=Q55PSEFHSAOGY7GD4DWI2G56U66BXVZHXZ52DX4LYV4B52F5ZZ6A, seg_pos={0 0}, upload=220713]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp): protocol: expected piece hash; context canceled; EOF Dec 16 03:12:21.014809482 (251h43m20.885892527s) [noise=true , address=77.90.31.10:4520 , piece_id=Q55PSEFHSAOGY7GD4DWI2G56U66BXVZHXZ52DX4LYV4B52F5ZZ6A, seg_pos={0 0}, upload=220713]: Long tail cancellation due to optimal threshold Dec 16 03:12:21.014826394 (251h43m20.885909429s) [seg_pos={0 0}, upload=220713]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 03:12:21.015234772 (251h43m20.886317807s) [upload=220713]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 03:12:21.015241204 (251h43m20.886324249s) [upload=220713]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 03:12:21.319403538 (251h43m21.190486583s) [upload=220713]: Done uploading segments... Dec 16 03:12:21.319427222 (251h43m21.190510257s) [upload=220713]: Upload finished. err: <nil> Duration: 6.000552968s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 5.686985089s trace: https://jaeger.dev.storj.io/trace/2e0c5260bf6208b3 FAIL (2025-12-16 03:13:35 UTC) orbiter warning: context deadline exceeded Duration: 5.68702166s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 5.784596167s trace: https://jaeger.dev.storj.io/trace/13f785634570d9fc FAIL (2025-12-16 03:13:24 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 16 03:12:18.507971969 (251h43m18.379055014s) [upload=220713]: Starting upload Dec 16 03:12:18.508025069 (251h43m18.379108115s) [upload=220713]: Done starting upload Dec 16 03:12:18.508197317 (251h43m18.379280353s) [upload=220713]: Uploading segments... Dec 16 03:12:18.508315022 (251h43m18.379398067s) [upload=220713]: Got next segment. Inline: false Dec 16 03:12:18.509349181 (251h43m18.380432216s) [upload=220713]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 03:12:18.509360643 (251h43m18.380443688s) [upload=220713]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 03:12:18.761879946 (251h43m18.632962991s) [seg_pos={0 0}, upload=220713]: Begin upload segment... Dec 16 03:12:18.761929029 (251h43m18.633012064s) [seg_pos={0 0}, upload=220713]: Done begin upload segment. Dec 16 03:12:18.761967552 (251h43m18.633050597s) [noise=true , address=77.90.31.10:4520 , piece_id=Q55PSEFHSAOGY7GD4DWI2G56U66BXVZHXZ52DX4LYV4B52F5ZZ6A, seg_pos={0 0}, upload=220713]: Uploading piece... Dec 16 03:12:18.762560491 (251h43m18.633643536s) [noise=true , address=45.91.116.6:4974 , piece_id=GQDHG4LFMNUHAKFA3GA3WANCW5OGHOWCYM3IZCUGTWI54LGFWDDA, seg_pos={0 0}, upload=220713]: Uploading piece... Dec 16 03:12:18.762660291 (251h43m18.633743336s) [noise=true , address=145.79.227.25:4617 , piece_id=TCLE2SK75NCWPLGUPYLJI3M5WD75Z2APQUGUXYJFFYAH5MU42T6A, seg_pos={0 0}, upload=220713]: Uploading piece... Dec 16 03:12:18.763036347 (251h43m18.634119382s) [noise=true , address=145.79.227.36:4518 , piece_id=I5C4LFPXUEB2JBUTWHXQV7VRJCCNN2RP7Z3CQ7RQENNMUBNNMHYQ, seg_pos={0 0}, upload=220713]: Uploading piece... Dec 16 03:12:18.763318324 (251h43m18.634401359s) [noise=true , address=145.79.227.14:4840 , piece_id=3ABV6MR5KUUYYF7I2TU5MEXF5ULEY6IU6YIF72DCGR5MBALRWF3A, seg_pos={0 0}, upload=220713]: Uploading piece... Dec 16 03:12:18.763607314 (251h43m18.634690349s) [noise=true , address=145.79.227.50:4373 , piece_id=YMG7Q5VQQKAYHO2HAEPOBRRDARQ25HGALNK3YOMIXUXYQONNKXSA, seg_pos={0 0}, upload=220713]: Uploading piece... Dec 16 03:12:18.763864645 (251h43m18.63494769s) [noise=true , address=145.79.227.51:4409 , piece_id=3IEX2COS67O2QHKMMPC2NZJ3PGDRDG3O5WOKIY5IA4HKMZFDNF4Q, seg_pos={0 0}, upload=220713]: Uploading piece... Dec 16 03:12:18.763888079 (251h43m18.634971154s) [noise=true , address=145.79.227.42:4123 , piece_id=NMJHDWWFESF3GM5TMEZFQGFD7YY5QHIIYQVHF3JIINSJHSEQDFZQ, seg_pos={0 0}, upload=220713]: Uploading piece... Dec 16 03:12:18.763947542 (251h43m18.635030607s) [noise=true , address=145.79.227.32:4834 , piece_id=RRMTYRQ37P6YWUV6BPVUC2BFLCRPWN6Q7VEVX6BDA54IKWTFH7SQ, seg_pos={0 0}, upload=220713]: Uploading piece... Dec 16 03:12:18.764263093 (251h43m18.635346138s) [noise=true , address=145.79.227.37:4565 , piece_id=4VFLWAOYZ7OOHBEDWSBLAWDZODVQEU3MFCQKSNGUE6HLC3MDPVCQ, seg_pos={0 0}, upload=220713]: Uploading piece... Dec 16 03:12:18.764390716 (251h43m18.635473761s) [noise=true , address=145.79.227.44:4194 , piece_id=RJPVGEHG2XIR5JAPDFN6FR5ZH7ICFU2ZQIQKURWSLTBQT4UV7POA, seg_pos={0 0}, upload=220713]: Uploading piece... Dec 16 03:12:18.764941455 (251h43m18.6360245s) [noise=true , address=145.79.227.33:4870 , piece_id=MRUDYCE63CU2SHIFP64BNHFRLYAO2A3KXN6GIZGG7QJGX6S7UXDA, seg_pos={0 0}, upload=220713]: Uploading piece... Dec 16 03:12:18.765504287 (251h43m18.636587322s) [noise=true , address=45.91.116.5:4298 , piece_id=GVZCYRIPTXYE6WOPQPRYKJSGV23I3GZ22SOCOJSXFUBLNDVQN2MA, seg_pos={0 0}, upload=220713]: Uploading piece... Dec 16 03:12:18.766127143 (251h43m18.637210188s) [noise=true , address=145.79.227.29:4942 , piece_id=SON2DHJWJHPLZ7IJG6HFLK6D6C5O6JL2DMLLPOB5GHSPET544HAA, seg_pos={0 0}, upload=220713]: Uploading piece... Dec 16 03:12:18.766432894 (251h43m18.637515929s) [noise=true , address=77.90.31.16:4735 , piece_id=ATZ3AJVXFRAXDM4DR6MKDEHTRHM6AG4RB4YLKDHDLDMO5XFF3MPA, seg_pos={0 0}, upload=220713]: Uploading piece... Dec 16 03:12:18.766736152 (251h43m18.637819187s) [noise=true , address=145.79.227.52:4447 , piece_id=OYGRYLI43OEMB7MBUTGLXG76U4WUUR3EMNCUZQ5C67DVHLWBVHJA, seg_pos={0 0}, upload=220713]: Uploading piece... Dec 16 03:12:20.470095008 (251h43m20.341178064s) [upload=220713]: Next returned nil segment Dec 16 03:12:20.470146787 (251h43m20.341229832s) [upload=220713]: Waiting for error group managing segments... Dec 16 03:12:20.753665772 (251h43m20.624748807s) [noise=true , address=145.79.227.42:4123 , piece_id=NMJHDWWFESF3GM5TMEZFQGFD7YY5QHIIYQVHF3JIINSJHSEQDFZQ, seg_pos={0 0}, upload=220713]: Done uploading piece. err: <nil> Dec 16 03:12:20.757729490 (251h43m20.628812535s) [noise=true , address=145.79.227.29:4942 , piece_id=SON2DHJWJHPLZ7IJG6HFLK6D6C5O6JL2DMLLPOB5GHSPET544HAA, seg_pos={0 0}, upload=220713]: Done uploading piece. err: <nil> Dec 16 03:12:20.760602501 (251h43m20.631685546s) [noise=true , address=145.79.227.36:4518 , piece_id=I5C4LFPXUEB2JBUTWHXQV7VRJCCNN2RP7Z3CQ7RQENNMUBNNMHYQ, seg_pos={0 0}, upload=220713]: Done uploading piece. err: <nil> Dec 16 03:12:20.764326061 (251h43m20.635409106s) [noise=true , address=145.79.227.25:4617 , piece_id=TCLE2SK75NCWPLGUPYLJI3M5WD75Z2APQUGUXYJFFYAH5MU42T6A, seg_pos={0 0}, upload=220713]: Done uploading piece. err: <nil> Dec 16 03:12:20.766406653 (251h43m20.637489698s) [noise=true , address=145.79.227.37:4565 , piece_id=4VFLWAOYZ7OOHBEDWSBLAWDZODVQEU3MFCQKSNGUE6HLC3MDPVCQ, seg_pos={0 0}, upload=220713]: Done uploading piece. err: <nil> Dec 16 03:12:20.770108361 (251h43m20.641191406s) [noise=true , address=145.79.227.33:4870 , piece_id=MRUDYCE63CU2SHIFP64BNHFRLYAO2A3KXN6GIZGG7QJGX6S7UXDA, seg_pos={0 0}, upload=220713]: Done uploading piece. err: <nil> Dec 16 03:12:20.773437491 (251h43m20.644520546s) [noise=true , address=145.79.227.50:4373 , piece_id=YMG7Q5VQQKAYHO2HAEPOBRRDARQ25HGALNK3YOMIXUXYQONNKXSA, seg_pos={0 0}, upload=220713]: Done uploading piece. err: <nil> Dec 16 03:12:20.775082453 (251h43m20.646165488s) [noise=true , address=145.79.227.52:4447 , piece_id=OYGRYLI43OEMB7MBUTGLXG76U4WUUR3EMNCUZQ5C67DVHLWBVHJA, seg_pos={0 0}, upload=220713]: Done uploading piece. err: <nil> Dec 16 03:12:20.833660625 (251h43m20.70474366s) [noise=true , address=145.79.227.51:4409 , piece_id=3IEX2COS67O2QHKMMPC2NZJ3PGDRDG3O5WOKIY5IA4HKMZFDNF4Q, seg_pos={0 0}, upload=220713]: Done uploading piece. err: <nil> Dec 16 03:12:20.844657545 (251h43m20.71574058s) [noise=true , address=145.79.227.14:4840 , piece_id=3ABV6MR5KUUYYF7I2TU5MEXF5ULEY6IU6YIF72DCGR5MBALRWF3A, seg_pos={0 0}, upload=220713]: Done uploading piece. err: <nil> Dec 16 03:12:20.859069218 (251h43m20.730152273s) [noise=true , address=145.79.227.32:4834 , piece_id=RRMTYRQ37P6YWUV6BPVUC2BFLCRPWN6Q7VEVX6BDA54IKWTFH7SQ, seg_pos={0 0}, upload=220713]: Done uploading piece. err: <nil> Dec 16 03:12:21.007293480 (251h43m20.878376515s) [noise=true , address=77.90.31.16:4735 , piece_id=ATZ3AJVXFRAXDM4DR6MKDEHTRHM6AG4RB4YLKDHDLDMO5XFF3MPA, seg_pos={0 0}, upload=220713]: Done uploading piece. err: <nil> Dec 16 03:12:21.014381907 (251h43m20.885464952s) [noise=true , address=45.91.116.5:4298 , piece_id=GVZCYRIPTXYE6WOPQPRYKJSGV23I3GZ22SOCOJSXFUBLNDVQN2MA, seg_pos={0 0}, upload=220713]: Done uploading piece. err: <nil> Dec 16 03:12:21.014403138 (251h43m20.885486173s) [seg_pos={0 0}, upload=220713]: Segment reached optimal threshold of 13 pieces. Dec 16 03:12:21.014711706 (251h43m20.885794741s) [noise=true , address=145.79.227.44:4194 , piece_id=RJPVGEHG2XIR5JAPDFN6FR5ZH7ICFU2ZQIQKURWSLTBQT4UV7POA, seg_pos={0 0}, upload=220713]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh): protocol: expected piece hash; context canceled; EOF Dec 16 03:12:21.014721354 (251h43m20.885804389s) [noise=true , address=145.79.227.44:4194 , piece_id=RJPVGEHG2XIR5JAPDFN6FR5ZH7ICFU2ZQIQKURWSLTBQT4UV7POA, seg_pos={0 0}, upload=220713]: Long tail cancellation due to optimal threshold Dec 16 03:12:21.014764125 (251h43m20.88584716s) [noise=true , address=45.91.116.6:4974 , piece_id=GQDHG4LFMNUHAKFA3GA3WANCW5OGHOWCYM3IZCUGTWI54LGFWDDA, seg_pos={0 0}, upload=220713]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: expected piece hash; context canceled; EOF Dec 16 03:12:21.014782741 (251h43m20.885865786s) [noise=true , address=45.91.116.6:4974 , piece_id=GQDHG4LFMNUHAKFA3GA3WANCW5OGHOWCYM3IZCUGTWI54LGFWDDA, seg_pos={0 0}, upload=220713]: Long tail cancellation due to optimal threshold Dec 16 03:12:21.014804422 (251h43m20.885887467s) [noise=true , address=77.90.31.10:4520 , piece_id=Q55PSEFHSAOGY7GD4DWI2G56U66BXVZHXZ52DX4LYV4B52F5ZZ6A, seg_pos={0 0}, upload=220713]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp): protocol: expected piece hash; context canceled; EOF Dec 16 03:12:21.014809482 (251h43m20.885892527s) [noise=true , address=77.90.31.10:4520 , piece_id=Q55PSEFHSAOGY7GD4DWI2G56U66BXVZHXZ52DX4LYV4B52F5ZZ6A, seg_pos={0 0}, upload=220713]: Long tail cancellation due to optimal threshold Dec 16 03:12:21.014826394 (251h43m20.885909429s) [seg_pos={0 0}, upload=220713]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 03:12:21.015234772 (251h43m20.886317807s) [upload=220713]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 03:12:21.015241204 (251h43m20.886324249s) [upload=220713]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 03:12:21.319403538 (251h43m21.190486583s) [upload=220713]: Done uploading segments... Dec 16 03:12:21.319427222 (251h43m21.190510257s) [upload=220713]: Upload finished. err: <nil> Duration: 5.78463477s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 5.663577809s trace: https://jaeger.dev.storj.io/trace/2e0c5260bf6208b3 FAIL (2025-12-16 03:13:16 UTC) orbiter warning: context deadline exceeded Duration: 5.66360431s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.002832211s trace: https://jaeger.dev.storj.io/trace/13f785634570d9fc FAIL (2025-12-16 03:12:47 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 16 03:12:18.507971969 (251h43m18.379055014s) [upload=220713]: Starting upload Dec 16 03:12:18.508025069 (251h43m18.379108115s) [upload=220713]: Done starting upload Dec 16 03:12:18.508197317 (251h43m18.379280353s) [upload=220713]: Uploading segments... Dec 16 03:12:18.508315022 (251h43m18.379398067s) [upload=220713]: Got next segment. Inline: false Dec 16 03:12:18.509349181 (251h43m18.380432216s) [upload=220713]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 03:12:18.509360643 (251h43m18.380443688s) [upload=220713]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 03:12:18.761879946 (251h43m18.632962991s) [seg_pos={0 0}, upload=220713]: Begin upload segment... Dec 16 03:12:18.761929029 (251h43m18.633012064s) [seg_pos={0 0}, upload=220713]: Done begin upload segment. Dec 16 03:12:18.761967552 (251h43m18.633050597s) [noise=true , address=77.90.31.10:4520 , piece_id=Q55PSEFHSAOGY7GD4DWI2G56U66BXVZHXZ52DX4LYV4B52F5ZZ6A, seg_pos={0 0}, upload=220713]: Uploading piece... Dec 16 03:12:18.762560491 (251h43m18.633643536s) [noise=true , address=45.91.116.6:4974 , piece_id=GQDHG4LFMNUHAKFA3GA3WANCW5OGHOWCYM3IZCUGTWI54LGFWDDA, seg_pos={0 0}, upload=220713]: Uploading piece... Dec 16 03:12:18.762660291 (251h43m18.633743336s) [noise=true , address=145.79.227.25:4617 , piece_id=TCLE2SK75NCWPLGUPYLJI3M5WD75Z2APQUGUXYJFFYAH5MU42T6A, seg_pos={0 0}, upload=220713]: Uploading piece... Dec 16 03:12:18.763036347 (251h43m18.634119382s) [noise=true , address=145.79.227.36:4518 , piece_id=I5C4LFPXUEB2JBUTWHXQV7VRJCCNN2RP7Z3CQ7RQENNMUBNNMHYQ, seg_pos={0 0}, upload=220713]: Uploading piece... Dec 16 03:12:18.763318324 (251h43m18.634401359s) [noise=true , address=145.79.227.14:4840 , piece_id=3ABV6MR5KUUYYF7I2TU5MEXF5ULEY6IU6YIF72DCGR5MBALRWF3A, seg_pos={0 0}, upload=220713]: Uploading piece... Dec 16 03:12:18.763607314 (251h43m18.634690349s) [noise=true , address=145.79.227.50:4373 , piece_id=YMG7Q5VQQKAYHO2HAEPOBRRDARQ25HGALNK3YOMIXUXYQONNKXSA, seg_pos={0 0}, upload=220713]: Uploading piece... Dec 16 03:12:18.763864645 (251h43m18.63494769s) [noise=true , address=145.79.227.51:4409 , piece_id=3IEX2COS67O2QHKMMPC2NZJ3PGDRDG3O5WOKIY5IA4HKMZFDNF4Q, seg_pos={0 0}, upload=220713]: Uploading piece... Dec 16 03:12:18.763888079 (251h43m18.634971154s) [noise=true , address=145.79.227.42:4123 , piece_id=NMJHDWWFESF3GM5TMEZFQGFD7YY5QHIIYQVHF3JIINSJHSEQDFZQ, seg_pos={0 0}, upload=220713]: Uploading piece... Dec 16 03:12:18.763947542 (251h43m18.635030607s) [noise=true , address=145.79.227.32:4834 , piece_id=RRMTYRQ37P6YWUV6BPVUC2BFLCRPWN6Q7VEVX6BDA54IKWTFH7SQ, seg_pos={0 0}, upload=220713]: Uploading piece... Dec 16 03:12:18.764263093 (251h43m18.635346138s) [noise=true , address=145.79.227.37:4565 , piece_id=4VFLWAOYZ7OOHBEDWSBLAWDZODVQEU3MFCQKSNGUE6HLC3MDPVCQ, seg_pos={0 0}, upload=220713]: Uploading piece... Dec 16 03:12:18.764390716 (251h43m18.635473761s) [noise=true , address=145.79.227.44:4194 , piece_id=RJPVGEHG2XIR5JAPDFN6FR5ZH7ICFU2ZQIQKURWSLTBQT4UV7POA, seg_pos={0 0}, upload=220713]: Uploading piece... Dec 16 03:12:18.764941455 (251h43m18.6360245s) [noise=true , address=145.79.227.33:4870 , piece_id=MRUDYCE63CU2SHIFP64BNHFRLYAO2A3KXN6GIZGG7QJGX6S7UXDA, seg_pos={0 0}, upload=220713]: Uploading piece... Dec 16 03:12:18.765504287 (251h43m18.636587322s) [noise=true , address=45.91.116.5:4298 , piece_id=GVZCYRIPTXYE6WOPQPRYKJSGV23I3GZ22SOCOJSXFUBLNDVQN2MA, seg_pos={0 0}, upload=220713]: Uploading piece... Dec 16 03:12:18.766127143 (251h43m18.637210188s) [noise=true , address=145.79.227.29:4942 , piece_id=SON2DHJWJHPLZ7IJG6HFLK6D6C5O6JL2DMLLPOB5GHSPET544HAA, seg_pos={0 0}, upload=220713]: Uploading piece... Dec 16 03:12:18.766432894 (251h43m18.637515929s) [noise=true , address=77.90.31.16:4735 , piece_id=ATZ3AJVXFRAXDM4DR6MKDEHTRHM6AG4RB4YLKDHDLDMO5XFF3MPA, seg_pos={0 0}, upload=220713]: Uploading piece... Dec 16 03:12:18.766736152 (251h43m18.637819187s) [noise=true , address=145.79.227.52:4447 , piece_id=OYGRYLI43OEMB7MBUTGLXG76U4WUUR3EMNCUZQ5C67DVHLWBVHJA, seg_pos={0 0}, upload=220713]: Uploading piece... Dec 16 03:12:20.470095008 (251h43m20.341178064s) [upload=220713]: Next returned nil segment Dec 16 03:12:20.470146787 (251h43m20.341229832s) [upload=220713]: Waiting for error group managing segments... Dec 16 03:12:20.753665772 (251h43m20.624748807s) [noise=true , address=145.79.227.42:4123 , piece_id=NMJHDWWFESF3GM5TMEZFQGFD7YY5QHIIYQVHF3JIINSJHSEQDFZQ, seg_pos={0 0}, upload=220713]: Done uploading piece. err: <nil> Dec 16 03:12:20.757729490 (251h43m20.628812535s) [noise=true , address=145.79.227.29:4942 , piece_id=SON2DHJWJHPLZ7IJG6HFLK6D6C5O6JL2DMLLPOB5GHSPET544HAA, seg_pos={0 0}, upload=220713]: Done uploading piece. err: <nil> Dec 16 03:12:20.760602501 (251h43m20.631685546s) [noise=true , address=145.79.227.36:4518 , piece_id=I5C4LFPXUEB2JBUTWHXQV7VRJCCNN2RP7Z3CQ7RQENNMUBNNMHYQ, seg_pos={0 0}, upload=220713]: Done uploading piece. err: <nil> Dec 16 03:12:20.764326061 (251h43m20.635409106s) [noise=true , address=145.79.227.25:4617 , piece_id=TCLE2SK75NCWPLGUPYLJI3M5WD75Z2APQUGUXYJFFYAH5MU42T6A, seg_pos={0 0}, upload=220713]: Done uploading piece. err: <nil> Dec 16 03:12:20.766406653 (251h43m20.637489698s) [noise=true , address=145.79.227.37:4565 , piece_id=4VFLWAOYZ7OOHBEDWSBLAWDZODVQEU3MFCQKSNGUE6HLC3MDPVCQ, seg_pos={0 0}, upload=220713]: Done uploading piece. err: <nil> Dec 16 03:12:20.770108361 (251h43m20.641191406s) [noise=true , address=145.79.227.33:4870 , piece_id=MRUDYCE63CU2SHIFP64BNHFRLYAO2A3KXN6GIZGG7QJGX6S7UXDA, seg_pos={0 0}, upload=220713]: Done uploading piece. err: <nil> Dec 16 03:12:20.773437491 (251h43m20.644520546s) [noise=true , address=145.79.227.50:4373 , piece_id=YMG7Q5VQQKAYHO2HAEPOBRRDARQ25HGALNK3YOMIXUXYQONNKXSA, seg_pos={0 0}, upload=220713]: Done uploading piece. err: <nil> Dec 16 03:12:20.775082453 (251h43m20.646165488s) [noise=true , address=145.79.227.52:4447 , piece_id=OYGRYLI43OEMB7MBUTGLXG76U4WUUR3EMNCUZQ5C67DVHLWBVHJA, seg_pos={0 0}, upload=220713]: Done uploading piece. err: <nil> Dec 16 03:12:20.833660625 (251h43m20.70474366s) [noise=true , address=145.79.227.51:4409 , piece_id=3IEX2COS67O2QHKMMPC2NZJ3PGDRDG3O5WOKIY5IA4HKMZFDNF4Q, seg_pos={0 0}, upload=220713]: Done uploading piece. err: <nil> Dec 16 03:12:20.844657545 (251h43m20.71574058s) [noise=true , address=145.79.227.14:4840 , piece_id=3ABV6MR5KUUYYF7I2TU5MEXF5ULEY6IU6YIF72DCGR5MBALRWF3A, seg_pos={0 0}, upload=220713]: Done uploading piece. err: <nil> Dec 16 03:12:20.859069218 (251h43m20.730152273s) [noise=true , address=145.79.227.32:4834 , piece_id=RRMTYRQ37P6YWUV6BPVUC2BFLCRPWN6Q7VEVX6BDA54IKWTFH7SQ, seg_pos={0 0}, upload=220713]: Done uploading piece. err: <nil> Dec 16 03:12:21.007293480 (251h43m20.878376515s) [noise=true , address=77.90.31.16:4735 , piece_id=ATZ3AJVXFRAXDM4DR6MKDEHTRHM6AG4RB4YLKDHDLDMO5XFF3MPA, seg_pos={0 0}, upload=220713]: Done uploading piece. err: <nil> Dec 16 03:12:21.014381907 (251h43m20.885464952s) [noise=true , address=45.91.116.5:4298 , piece_id=GVZCYRIPTXYE6WOPQPRYKJSGV23I3GZ22SOCOJSXFUBLNDVQN2MA, seg_pos={0 0}, upload=220713]: Done uploading piece. err: <nil> Dec 16 03:12:21.014403138 (251h43m20.885486173s) [seg_pos={0 0}, upload=220713]: Segment reached optimal threshold of 13 pieces. Dec 16 03:12:21.014711706 (251h43m20.885794741s) [noise=true , address=145.79.227.44:4194 , piece_id=RJPVGEHG2XIR5JAPDFN6FR5ZH7ICFU2ZQIQKURWSLTBQT4UV7POA, seg_pos={0 0}, upload=220713]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh): protocol: expected piece hash; context canceled; EOF Dec 16 03:12:21.014721354 (251h43m20.885804389s) [noise=true , address=145.79.227.44:4194 , piece_id=RJPVGEHG2XIR5JAPDFN6FR5ZH7ICFU2ZQIQKURWSLTBQT4UV7POA, seg_pos={0 0}, upload=220713]: Long tail cancellation due to optimal threshold Dec 16 03:12:21.014764125 (251h43m20.88584716s) [noise=true , address=45.91.116.6:4974 , piece_id=GQDHG4LFMNUHAKFA3GA3WANCW5OGHOWCYM3IZCUGTWI54LGFWDDA, seg_pos={0 0}, upload=220713]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: expected piece hash; context canceled; EOF Dec 16 03:12:21.014782741 (251h43m20.885865786s) [noise=true , address=45.91.116.6:4974 , piece_id=GQDHG4LFMNUHAKFA3GA3WANCW5OGHOWCYM3IZCUGTWI54LGFWDDA, seg_pos={0 0}, upload=220713]: Long tail cancellation due to optimal threshold Dec 16 03:12:21.014804422 (251h43m20.885887467s) [noise=true , address=77.90.31.10:4520 , piece_id=Q55PSEFHSAOGY7GD4DWI2G56U66BXVZHXZ52DX4LYV4B52F5ZZ6A, seg_pos={0 0}, upload=220713]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp): protocol: expected piece hash; context canceled; EOF Dec 16 03:12:21.014809482 (251h43m20.885892527s) [noise=true , address=77.90.31.10:4520 , piece_id=Q55PSEFHSAOGY7GD4DWI2G56U66BXVZHXZ52DX4LYV4B52F5ZZ6A, seg_pos={0 0}, upload=220713]: Long tail cancellation due to optimal threshold Dec 16 03:12:21.014826394 (251h43m20.885909429s) [seg_pos={0 0}, upload=220713]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 03:12:21.015234772 (251h43m20.886317807s) [upload=220713]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 03:12:21.015241204 (251h43m20.886324249s) [upload=220713]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 03:12:21.319403538 (251h43m21.190486583s) [upload=220713]: Done uploading segments... Dec 16 03:12:21.319427222 (251h43m21.190510257s) [upload=220713]: Upload finished. err: <nil> Duration: 30.002856478s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 5.841433583s trace: https://jaeger.dev.storj.io/trace/2e0c5260bf6208b3 FAIL (2025-12-16 03:12:26 UTC) orbiter warning: context deadline exceeded Duration: 5.84145259s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 5.998205026s trace: https://jaeger.dev.storj.io/trace/2e0c5260bf6208b3 FAIL (2025-12-16 03:11:57 UTC) orbiter warning: context deadline exceeded Duration: 5.998232779s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 5.974549677s trace: https://jaeger.dev.storj.io/trace/2e0c5260bf6208b3 FAIL (2025-12-16 03:10:47 UTC) orbiter warning: context deadline exceeded Duration: 5.974626084s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 6.054442523s trace: https://jaeger.dev.storj.io/trace/2e0c5260bf6208b3 FAIL (2025-12-16 03:09:38 UTC) orbiter warning: context deadline exceeded Duration: 6.054467622s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 5.854372641s trace: https://jaeger.dev.storj.io/trace/2e0c5260bf6208b3 FAIL (2025-12-16 03:08:34 UTC) orbiter warning: context deadline exceeded Duration: 5.854400383s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 5.810264079s trace: https://jaeger.dev.storj.io/trace/2e0c5260bf6208b3 FAIL (2025-12-16 03:07:47 UTC) orbiter warning: context deadline exceeded Duration: 5.810290048s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 6.01564289s trace: https://jaeger.dev.storj.io/trace/2e0c5260bf6208b3 FAIL (2025-12-16 03:07:17 UTC) orbiter warning: context deadline exceeded Duration: 6.015666235s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 5.620725276s trace: https://jaeger.dev.storj.io/trace/2e0c5260bf6208b3 FAIL (2025-12-16 03:06:55 UTC) orbiter warning: context deadline exceeded Duration: 5.62075892s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 6.153273901s trace: https://jaeger.dev.storj.io/trace/2e0c5260bf6208b3 FAIL (2025-12-16 03:05:43 UTC) orbiter warning: context deadline exceeded Duration: 6.153300022s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 5.956087445s trace: https://jaeger.dev.storj.io/trace/2e0c5260bf6208b3 FAIL (2025-12-16 03:05:10 UTC) orbiter warning: context deadline exceeded Duration: 5.956117161s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 6.709111233s trace: https://jaeger.dev.storj.io/trace/2e0c5260bf6208b3 FAIL (2025-12-16 03:04:29 UTC) orbiter warning: context deadline exceeded Duration: 6.709142612s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 6.104929006s trace: https://jaeger.dev.storj.io/trace/2e0c5260bf6208b3 FAIL (2025-12-16 03:04:11 UTC) orbiter warning: context deadline exceeded Duration: 6.104949546s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 1m0.001485538s trace: https://jaeger.dev.storj.io/trace/2e0c5260bf6208b3 FAIL (2025-12-16 03:04:04 UTC) orbiter warning: context deadline exceeded Duration: 1m0.001510124s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [X...........................................................] duration: 3.516690459s trace: https://jaeger.dev.storj.io/trace/44e6044b9b6d58a0 FAIL (2025-12-15 23:43:27 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.516718583s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [XX..........................................................] duration: 3.613049686s trace: https://jaeger.dev.storj.io/trace/44e6044b9b6d58a0 FAIL (2025-12-15 23:42:21 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.613131592s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [XXX.........................................................] duration: 3.519550687s trace: https://jaeger.dev.storj.io/trace/44e6044b9b6d58a0 FAIL (2025-12-15 23:41:56 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.51957868s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [XXXX........................................................] duration: 3.513255708s trace: https://jaeger.dev.storj.io/trace/44e6044b9b6d58a0 FAIL (2025-12-15 23:41:36 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.513282328s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 5 out of 60 is failed (threshold=6) [XXXXX.......................................................] duration: 3.615469868s trace: https://jaeger.dev.storj.io/trace/44e6044b9b6d58a0 FAIL (2025-12-15 23:40:41 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.615514904s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [XXXXXX......................................................] duration: 3.55960514s trace: https://jaeger.dev.storj.io/trace/44e6044b9b6d58a0 FAIL (2025-12-15 23:39:52 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.559629677s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 7 out of 60 is failed (threshold=6) [XXXXXXX.....................................................] duration: 3.455033535s trace: https://jaeger.dev.storj.io/trace/44e6044b9b6d58a0 FAIL (2025-12-15 23:39:17 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.455064686s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 8 out of 60 is failed (threshold=6) [XXXXXXXX....................................................] duration: 3.601754323s trace: https://jaeger.dev.storj.io/trace/44e6044b9b6d58a0 FAIL (2025-12-15 23:38:59 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.601778258s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 9 out of 60 is failed (threshold=6) [XXXXXXXXX...................................................] duration: 3.544240687s trace: https://jaeger.dev.storj.io/trace/44e6044b9b6d58a0 FAIL (2025-12-15 23:38:34 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.544269571s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)