Orbiter History

Orbiter: EU/US/AP/EDGE | Code | Config | Playbook | Escalation Procedure

Orbiter Check History

Showing 50 of 866735 reports (Page 11650 of 17335)

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [...................................X.............X.......XX.] duration: 3.552375367s trace: https://jaeger.dev.storj.io/trace/732a0ad34bfabe6a FAIL (2025-12-09 02:00:18 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.552407868s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 6.082316096s trace: https://jaeger.dev.storj.io/trace/1b9ac1acdb153d00 FAIL (2025-12-09 02:00:03 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== A, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.018287744 (81h49m44.88937079s) [noise=true , address=45.91.116.10:4098 , piece_id=Q6MKNY37MMBVHUOMWJCTPE5BPXWG7G7RLEHQKLXYSDSI2KKYTPNQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.018979372 (81h49m44.890062467s) [noise=true , address=45.91.116.5:4306 , piece_id=5S4PWZULVPGCP4I4PSINBYKIOCBPLFIUOZE5QO4CKTQ3I7P76BQQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.019582951 (81h49m44.890666006s) [noise=true , address=77.90.31.10:4538 , piece_id=KTOSBWL7QVBHLWCAMOZNTHRCLWMUUHCLIPI2W52KOHNJ55NLNRWQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.019819150 (81h49m44.890902196s) [noise=true , address=77.90.31.10:4530 , piece_id=XVCYKZOJ2H3U5RWMWW3BIBW37YK3ZQUHNIJHSGGTP7R2X244Y55Q, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.020245322 (81h49m44.891328377s) [noise=true , address=45.91.116.9:4287 , piece_id=GUUNM5MV3J7OELO633ESY3EMGLJOZKXNUKT3SRI7GKYJDYIC6SPA, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.020660623 (81h49m44.891743658s) [noise=true , address=145.79.227.12:4769 , piece_id=YTUOFFOTJBPKJUKCRDIY6S65ZP3W5XJCW7GU6GLSFHDUOPOTQO4Q, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.021074250 (81h49m44.892157295s) [noise=true , address=77.90.31.11:4570 , piece_id=HIPMDO4ST4XFQDT3M7PFIIARDMPBX2WEIPRVHOQI4OEIPMJIZZRA, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.021462369 (81h49m44.892545414s) [noise=true , address=145.79.227.49:4359 , piece_id=FDOP6PX5XFB5OJVC7H4S6ZFY4FT5AAWOLIH43M4GTLCZS5YKIM5Q, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.021981778 (81h49m44.893064813s) [noise=true , address=145.79.227.52:4454 , piece_id=7YBPVY5EXGAIQKJEP5TCXX5KRFBUBC45QQI6CONCS5LGUMRU3Y4Q, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.022307669 (81h49m44.893390704s) [noise=true , address=145.79.227.9:4384 , piece_id=3JXASIXRKWJ573Q466LHQLNOZXF4UC2A6LRJKZE2KBOY2N444EUA, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.022633470 (81h49m44.893716505s) [noise=true , address=64.49.13.170:4722 , piece_id=Y5ZZFYYBICQCKHXREFINQ6LA6IUU4MAS4OA4SJXFBC2BFMIG4IMQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.023012681 (81h49m44.894095716s) [noise=true , address=77.90.31.16:4755 , piece_id=M2EBCSYLP54HIYVMV2WFXQRTLSISUJKZPGQRB6K6EEWSRKJKWVYA, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.023282415 (81h49m44.89436546s) [noise=true , address=145.79.227.4:4205 , piece_id=BWNYAT67IELSPLQTYMAM2O76OLQWUTXACQXR4VZSAQE67NYOASRA, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.023555585 (81h49m44.89463862s) [noise=true , address=77.90.31.16:4736 , piece_id=BHZSQ7MXD2X4YEGL6IYMUYCHURUXDRRRJDYPTBVZ44SORXMPSO5A, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.023817675 (81h49m44.89490071s) [noise=true , address=77.90.31.15:4713 , piece_id=BCW7S3OAKKGDYPH7RP7B6VJJSG5HNCR2DTLMHZDRLO4EEOPH7FKQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.024446281 (81h49m44.895529327s) [noise=true , address=77.90.31.17:4769 , piece_id=66IBD3RPUO7QFO4ALJSH5W4ELFKPLDOHSMIQ7UWNGGDV3OXZGUQQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.024703942 (81h49m44.895786987s) [noise=true , address=145.79.227.39:4037 , piece_id=PGBSSKDC7LGLKZE3FGX6FHAC6H45RMOLJFVQHCJPWWAVZSOSIG5A, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:46.437570978 (81h49m46.308654013s) [upload=71502]: Next returned nil segment Dec 9 01:18:46.437586438 (81h49m46.308669483s) [upload=71502]: Waiting for error group managing segments... Dec 9 01:18:46.796243098 (81h49m46.667326134s) [noise=true , address=64.49.13.170:4722 , piece_id=Y5ZZFYYBICQCKHXREFINQ6LA6IUU4MAS4OA4SJXFBC2BFMIG4IMQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.894539060 (81h49m46.765622105s) [noise=true , address=145.79.227.39:4037 , piece_id=PGBSSKDC7LGLKZE3FGX6FHAC6H45RMOLJFVQHCJPWWAVZSOSIG5A, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.906718262 (81h49m46.777801297s) [noise=true , address=145.79.227.9:4384 , piece_id=3JXASIXRKWJ573Q466LHQLNOZXF4UC2A6LRJKZE2KBOY2N444EUA, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.914490982 (81h49m46.785574027s) [noise=true , address=145.79.227.49:4359 , piece_id=FDOP6PX5XFB5OJVC7H4S6ZFY4FT5AAWOLIH43M4GTLCZS5YKIM5Q, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.928206178 (81h49m46.799289213s) [noise=true , address=145.79.227.52:4454 , piece_id=7YBPVY5EXGAIQKJEP5TCXX5KRFBUBC45QQI6CONCS5LGUMRU3Y4Q, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.953023966 (81h49m46.824107001s) [noise=true , address=145.79.227.12:4769 , piece_id=YTUOFFOTJBPKJUKCRDIY6S65ZP3W5XJCW7GU6GLSFHDUOPOTQO4Q, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.994863524 (81h49m46.865946569s) [noise=true , address=77.90.31.16:4755 , piece_id=M2EBCSYLP54HIYVMV2WFXQRTLSISUJKZPGQRB6K6EEWSRKJKWVYA, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.998843783 (81h49m46.869926818s) [noise=true , address=77.90.31.18:4830 , piece_id=3CTHRQHH7RNBOS6DDU3PUHE5TGEQMB4UBDSJEXE67GM7PXJIEO4A, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.002758567 (81h49m46.873841602s) [noise=true , address=77.90.31.17:4769 , piece_id=66IBD3RPUO7QFO4ALJSH5W4ELFKPLDOHSMIQ7UWNGGDV3OXZGUQQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.010756346 (81h49m46.881839382s) [noise=true , address=77.90.31.11:4570 , piece_id=HIPMDO4ST4XFQDT3M7PFIIARDMPBX2WEIPRVHOQI4OEIPMJIZZRA, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.088053593 (81h49m46.959136638s) [noise=true , address=45.91.116.10:4098 , piece_id=Q6MKNY37MMBVHUOMWJCTPE5BPXWG7G7RLEHQKLXYSDSI2KKYTPNQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.094063277 (81h49m46.965146322s) [noise=true , address=77.90.31.10:4530 , piece_id=XVCYKZOJ2H3U5RWMWW3BIBW37YK3ZQUHNIJHSGGTP7R2X244Y55Q, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.101953681 (81h49m46.973036726s) [noise=true , address=45.91.116.5:4306 , piece_id=5S4PWZULVPGCP4I4PSINBYKIOCBPLFIUOZE5QO4CKTQ3I7P76BQQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.109207003 (81h49m46.980290038s) [noise=true , address=77.90.31.10:4538 , piece_id=KTOSBWL7QVBHLWCAMOZNTHRCLWMUUHCLIPI2W52KOHNJ55NLNRWQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.109225769 (81h49m46.980308804s) [seg_pos={0 0}, upload=71502]: Segment reached optimal threshold of 14 pieces. Dec 9 01:18:47.109408547 (81h49m46.980491592s) [noise=true , address=77.90.31.15:4713 , piece_id=BCW7S3OAKKGDYPH7RP7B6VJJSG5HNCR2DTLMHZDRLO4EEOPH7FKQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Liv9RcEgfjxodPuadijzkcUGgJEy7LfKqHGFhaJCKbcxKkNN8): protocol: expected piece hash; context canceled; EOF Dec 9 01:18:47.109416322 (81h49m46.980499357s) [noise=true , address=77.90.31.15:4713 , piece_id=BCW7S3OAKKGDYPH7RP7B6VJJSG5HNCR2DTLMHZDRLO4EEOPH7FKQ, seg_pos={0 0}, upload=71502]: Long tail cancellation due to optimal threshold Dec 9 01:18:47.109443945 (81h49m46.98052698s) [noise=true , address=45.91.116.9:4287 , piece_id=GUUNM5MV3J7OELO633ESY3EMGLJOZKXNUKT3SRI7GKYJDYIC6SPA, seg_pos={0 0}, upload=71502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cNuooGmeprkK8fRRBY5ZSAbWda7fdfPbYwbg3fV2TKGsYikwf): protocol: expected piece hash; context canceled; EOF Dec 9 01:18:47.109452361 (81h49m46.980535396s) [noise=true , address=45.91.116.9:4287 , piece_id=GUUNM5MV3J7OELO633ESY3EMGLJOZKXNUKT3SRI7GKYJDYIC6SPA, seg_pos={0 0}, upload=71502]: Long tail cancellation due to optimal threshold Dec 9 01:18:47.109521943 (81h49m46.980604978s) [noise=true , address=145.79.227.4:4205 , piece_id=BWNYAT67IELSPLQTYMAM2O76OLQWUTXACQXR4VZSAQE67NYOASRA, seg_pos={0 0}, upload=71502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pih2cgjPo5YeP6B7g8diM7p11VRJfHy5H34GUT9CK77W2cTLr): protocol: expected piece hash; context canceled; EOF Dec 9 01:18:47.109527924 (81h49m46.980610969s) [noise=true , address=145.79.227.4:4205 , piece_id=BWNYAT67IELSPLQTYMAM2O76OLQWUTXACQXR4VZSAQE67NYOASRA, seg_pos={0 0}, upload=71502]: Long tail cancellation due to optimal threshold Dec 9 01:18:47.109629788 (81h49m46.980712823s) [noise=true , address=77.90.31.16:4736 , piece_id=BHZSQ7MXD2X4YEGL6IYMUYCHURUXDRRRJDYPTBVZ44SORXMPSO5A, seg_pos={0 0}, upload=71502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: expected piece hash; context canceled; EOF Dec 9 01:18:47.109635399 (81h49m46.980718444s) [noise=true , address=77.90.31.16:4736 , piece_id=BHZSQ7MXD2X4YEGL6IYMUYCHURUXDRRRJDYPTBVZ44SORXMPSO5A, seg_pos={0 0}, upload=71502]: Long tail cancellation due to optimal threshold Dec 9 01:18:47.109669504 (81h49m46.980752549s) [seg_pos={0 0}, upload=71502]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 9 01:18:47.109759546 (81h49m46.980842591s) [upload=71502]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 01:18:47.109766839 (81h49m46.980849874s) [upload=71502]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 01:19:58.768357796 (81h50m58.639440831s) [upload=71502]: Done uploading segments... Dec 9 01:19:58.768409825 (81h50m58.63949286s) [upload=71502]: Upload finished. err: metaclient: context deadline exceeded Duration: 6.08233885s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [.X............X.X.........................................X.] duration: 5.736148165s trace: https://jaeger.dev.storj.io/trace/24d2447efd4b070b FAIL (2025-12-09 01:59:42 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.736163374s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 2.014865534s trace: https://jaeger.dev.storj.io/trace/36455c540a91ba90 FAIL (2025-12-09 01:59:39 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 2.01489493s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [..X............X.X.........................................X] duration: 1m0.001011415s trace: https://jaeger.dev.storj.io/trace/24d2447efd4b070b FAIL (2025-12-09 01:59:36 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.001032815s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [....................................X.............X.......XX] duration: 5.001676416s trace: https://jaeger.dev.storj.io/trace/732a0ad34bfabe6a FAIL (2025-12-09 01:59:20 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001695242s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 5.860347152s trace: https://jaeger.dev.storj.io/trace/1b9ac1acdb153d00 FAIL (2025-12-09 01:59:15 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== A, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.018287744 (81h49m44.88937079s) [noise=true , address=45.91.116.10:4098 , piece_id=Q6MKNY37MMBVHUOMWJCTPE5BPXWG7G7RLEHQKLXYSDSI2KKYTPNQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.018979372 (81h49m44.890062467s) [noise=true , address=45.91.116.5:4306 , piece_id=5S4PWZULVPGCP4I4PSINBYKIOCBPLFIUOZE5QO4CKTQ3I7P76BQQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.019582951 (81h49m44.890666006s) [noise=true , address=77.90.31.10:4538 , piece_id=KTOSBWL7QVBHLWCAMOZNTHRCLWMUUHCLIPI2W52KOHNJ55NLNRWQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.019819150 (81h49m44.890902196s) [noise=true , address=77.90.31.10:4530 , piece_id=XVCYKZOJ2H3U5RWMWW3BIBW37YK3ZQUHNIJHSGGTP7R2X244Y55Q, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.020245322 (81h49m44.891328377s) [noise=true , address=45.91.116.9:4287 , piece_id=GUUNM5MV3J7OELO633ESY3EMGLJOZKXNUKT3SRI7GKYJDYIC6SPA, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.020660623 (81h49m44.891743658s) [noise=true , address=145.79.227.12:4769 , piece_id=YTUOFFOTJBPKJUKCRDIY6S65ZP3W5XJCW7GU6GLSFHDUOPOTQO4Q, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.021074250 (81h49m44.892157295s) [noise=true , address=77.90.31.11:4570 , piece_id=HIPMDO4ST4XFQDT3M7PFIIARDMPBX2WEIPRVHOQI4OEIPMJIZZRA, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.021462369 (81h49m44.892545414s) [noise=true , address=145.79.227.49:4359 , piece_id=FDOP6PX5XFB5OJVC7H4S6ZFY4FT5AAWOLIH43M4GTLCZS5YKIM5Q, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.021981778 (81h49m44.893064813s) [noise=true , address=145.79.227.52:4454 , piece_id=7YBPVY5EXGAIQKJEP5TCXX5KRFBUBC45QQI6CONCS5LGUMRU3Y4Q, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.022307669 (81h49m44.893390704s) [noise=true , address=145.79.227.9:4384 , piece_id=3JXASIXRKWJ573Q466LHQLNOZXF4UC2A6LRJKZE2KBOY2N444EUA, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.022633470 (81h49m44.893716505s) [noise=true , address=64.49.13.170:4722 , piece_id=Y5ZZFYYBICQCKHXREFINQ6LA6IUU4MAS4OA4SJXFBC2BFMIG4IMQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.023012681 (81h49m44.894095716s) [noise=true , address=77.90.31.16:4755 , piece_id=M2EBCSYLP54HIYVMV2WFXQRTLSISUJKZPGQRB6K6EEWSRKJKWVYA, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.023282415 (81h49m44.89436546s) [noise=true , address=145.79.227.4:4205 , piece_id=BWNYAT67IELSPLQTYMAM2O76OLQWUTXACQXR4VZSAQE67NYOASRA, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.023555585 (81h49m44.89463862s) [noise=true , address=77.90.31.16:4736 , piece_id=BHZSQ7MXD2X4YEGL6IYMUYCHURUXDRRRJDYPTBVZ44SORXMPSO5A, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.023817675 (81h49m44.89490071s) [noise=true , address=77.90.31.15:4713 , piece_id=BCW7S3OAKKGDYPH7RP7B6VJJSG5HNCR2DTLMHZDRLO4EEOPH7FKQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.024446281 (81h49m44.895529327s) [noise=true , address=77.90.31.17:4769 , piece_id=66IBD3RPUO7QFO4ALJSH5W4ELFKPLDOHSMIQ7UWNGGDV3OXZGUQQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.024703942 (81h49m44.895786987s) [noise=true , address=145.79.227.39:4037 , piece_id=PGBSSKDC7LGLKZE3FGX6FHAC6H45RMOLJFVQHCJPWWAVZSOSIG5A, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:46.437570978 (81h49m46.308654013s) [upload=71502]: Next returned nil segment Dec 9 01:18:46.437586438 (81h49m46.308669483s) [upload=71502]: Waiting for error group managing segments... Dec 9 01:18:46.796243098 (81h49m46.667326134s) [noise=true , address=64.49.13.170:4722 , piece_id=Y5ZZFYYBICQCKHXREFINQ6LA6IUU4MAS4OA4SJXFBC2BFMIG4IMQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.894539060 (81h49m46.765622105s) [noise=true , address=145.79.227.39:4037 , piece_id=PGBSSKDC7LGLKZE3FGX6FHAC6H45RMOLJFVQHCJPWWAVZSOSIG5A, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.906718262 (81h49m46.777801297s) [noise=true , address=145.79.227.9:4384 , piece_id=3JXASIXRKWJ573Q466LHQLNOZXF4UC2A6LRJKZE2KBOY2N444EUA, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.914490982 (81h49m46.785574027s) [noise=true , address=145.79.227.49:4359 , piece_id=FDOP6PX5XFB5OJVC7H4S6ZFY4FT5AAWOLIH43M4GTLCZS5YKIM5Q, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.928206178 (81h49m46.799289213s) [noise=true , address=145.79.227.52:4454 , piece_id=7YBPVY5EXGAIQKJEP5TCXX5KRFBUBC45QQI6CONCS5LGUMRU3Y4Q, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.953023966 (81h49m46.824107001s) [noise=true , address=145.79.227.12:4769 , piece_id=YTUOFFOTJBPKJUKCRDIY6S65ZP3W5XJCW7GU6GLSFHDUOPOTQO4Q, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.994863524 (81h49m46.865946569s) [noise=true , address=77.90.31.16:4755 , piece_id=M2EBCSYLP54HIYVMV2WFXQRTLSISUJKZPGQRB6K6EEWSRKJKWVYA, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.998843783 (81h49m46.869926818s) [noise=true , address=77.90.31.18:4830 , piece_id=3CTHRQHH7RNBOS6DDU3PUHE5TGEQMB4UBDSJEXE67GM7PXJIEO4A, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.002758567 (81h49m46.873841602s) [noise=true , address=77.90.31.17:4769 , piece_id=66IBD3RPUO7QFO4ALJSH5W4ELFKPLDOHSMIQ7UWNGGDV3OXZGUQQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.010756346 (81h49m46.881839382s) [noise=true , address=77.90.31.11:4570 , piece_id=HIPMDO4ST4XFQDT3M7PFIIARDMPBX2WEIPRVHOQI4OEIPMJIZZRA, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.088053593 (81h49m46.959136638s) [noise=true , address=45.91.116.10:4098 , piece_id=Q6MKNY37MMBVHUOMWJCTPE5BPXWG7G7RLEHQKLXYSDSI2KKYTPNQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.094063277 (81h49m46.965146322s) [noise=true , address=77.90.31.10:4530 , piece_id=XVCYKZOJ2H3U5RWMWW3BIBW37YK3ZQUHNIJHSGGTP7R2X244Y55Q, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.101953681 (81h49m46.973036726s) [noise=true , address=45.91.116.5:4306 , piece_id=5S4PWZULVPGCP4I4PSINBYKIOCBPLFIUOZE5QO4CKTQ3I7P76BQQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.109207003 (81h49m46.980290038s) [noise=true , address=77.90.31.10:4538 , piece_id=KTOSBWL7QVBHLWCAMOZNTHRCLWMUUHCLIPI2W52KOHNJ55NLNRWQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.109225769 (81h49m46.980308804s) [seg_pos={0 0}, upload=71502]: Segment reached optimal threshold of 14 pieces. Dec 9 01:18:47.109408547 (81h49m46.980491592s) [noise=true , address=77.90.31.15:4713 , piece_id=BCW7S3OAKKGDYPH7RP7B6VJJSG5HNCR2DTLMHZDRLO4EEOPH7FKQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Liv9RcEgfjxodPuadijzkcUGgJEy7LfKqHGFhaJCKbcxKkNN8): protocol: expected piece hash; context canceled; EOF Dec 9 01:18:47.109416322 (81h49m46.980499357s) [noise=true , address=77.90.31.15:4713 , piece_id=BCW7S3OAKKGDYPH7RP7B6VJJSG5HNCR2DTLMHZDRLO4EEOPH7FKQ, seg_pos={0 0}, upload=71502]: Long tail cancellation due to optimal threshold Dec 9 01:18:47.109443945 (81h49m46.98052698s) [noise=true , address=45.91.116.9:4287 , piece_id=GUUNM5MV3J7OELO633ESY3EMGLJOZKXNUKT3SRI7GKYJDYIC6SPA, seg_pos={0 0}, upload=71502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cNuooGmeprkK8fRRBY5ZSAbWda7fdfPbYwbg3fV2TKGsYikwf): protocol: expected piece hash; context canceled; EOF Dec 9 01:18:47.109452361 (81h49m46.980535396s) [noise=true , address=45.91.116.9:4287 , piece_id=GUUNM5MV3J7OELO633ESY3EMGLJOZKXNUKT3SRI7GKYJDYIC6SPA, seg_pos={0 0}, upload=71502]: Long tail cancellation due to optimal threshold Dec 9 01:18:47.109521943 (81h49m46.980604978s) [noise=true , address=145.79.227.4:4205 , piece_id=BWNYAT67IELSPLQTYMAM2O76OLQWUTXACQXR4VZSAQE67NYOASRA, seg_pos={0 0}, upload=71502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pih2cgjPo5YeP6B7g8diM7p11VRJfHy5H34GUT9CK77W2cTLr): protocol: expected piece hash; context canceled; EOF Dec 9 01:18:47.109527924 (81h49m46.980610969s) [noise=true , address=145.79.227.4:4205 , piece_id=BWNYAT67IELSPLQTYMAM2O76OLQWUTXACQXR4VZSAQE67NYOASRA, seg_pos={0 0}, upload=71502]: Long tail cancellation due to optimal threshold Dec 9 01:18:47.109629788 (81h49m46.980712823s) [noise=true , address=77.90.31.16:4736 , piece_id=BHZSQ7MXD2X4YEGL6IYMUYCHURUXDRRRJDYPTBVZ44SORXMPSO5A, seg_pos={0 0}, upload=71502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: expected piece hash; context canceled; EOF Dec 9 01:18:47.109635399 (81h49m46.980718444s) [noise=true , address=77.90.31.16:4736 , piece_id=BHZSQ7MXD2X4YEGL6IYMUYCHURUXDRRRJDYPTBVZ44SORXMPSO5A, seg_pos={0 0}, upload=71502]: Long tail cancellation due to optimal threshold Dec 9 01:18:47.109669504 (81h49m46.980752549s) [seg_pos={0 0}, upload=71502]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 9 01:18:47.109759546 (81h49m46.980842591s) [upload=71502]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 01:18:47.109766839 (81h49m46.980849874s) [upload=71502]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 01:19:58.768357796 (81h50m58.639440831s) [upload=71502]: Done uploading segments... Dec 9 01:19:58.768409825 (81h50m58.63949286s) [upload=71502]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.860385325s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 1.984387336s trace: https://jaeger.dev.storj.io/trace/36455c540a91ba90 FAIL (2025-12-09 01:58:51 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 1.984411882s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [.....................................X.............X.......X] duration: 5.000876163s trace: https://jaeger.dev.storj.io/trace/5d21e50c7335bf56 FAIL (2025-12-09 01:58:20 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.00089521s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 2.010298427s trace: https://jaeger.dev.storj.io/trace/36455c540a91ba90 FAIL (2025-12-09 01:58:19 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 2.010326571s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [...X............X.X.........................................] duration: 5.570571501s trace: https://jaeger.dev.storj.io/trace/4a1f040f80b86635 FAIL (2025-12-09 01:58:17 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.570645273s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 5.814826702s trace: https://jaeger.dev.storj.io/trace/1b9ac1acdb153d00 FAIL (2025-12-09 01:58:06 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== A, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.018287744 (81h49m44.88937079s) [noise=true , address=45.91.116.10:4098 , piece_id=Q6MKNY37MMBVHUOMWJCTPE5BPXWG7G7RLEHQKLXYSDSI2KKYTPNQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.018979372 (81h49m44.890062467s) [noise=true , address=45.91.116.5:4306 , piece_id=5S4PWZULVPGCP4I4PSINBYKIOCBPLFIUOZE5QO4CKTQ3I7P76BQQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.019582951 (81h49m44.890666006s) [noise=true , address=77.90.31.10:4538 , piece_id=KTOSBWL7QVBHLWCAMOZNTHRCLWMUUHCLIPI2W52KOHNJ55NLNRWQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.019819150 (81h49m44.890902196s) [noise=true , address=77.90.31.10:4530 , piece_id=XVCYKZOJ2H3U5RWMWW3BIBW37YK3ZQUHNIJHSGGTP7R2X244Y55Q, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.020245322 (81h49m44.891328377s) [noise=true , address=45.91.116.9:4287 , piece_id=GUUNM5MV3J7OELO633ESY3EMGLJOZKXNUKT3SRI7GKYJDYIC6SPA, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.020660623 (81h49m44.891743658s) [noise=true , address=145.79.227.12:4769 , piece_id=YTUOFFOTJBPKJUKCRDIY6S65ZP3W5XJCW7GU6GLSFHDUOPOTQO4Q, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.021074250 (81h49m44.892157295s) [noise=true , address=77.90.31.11:4570 , piece_id=HIPMDO4ST4XFQDT3M7PFIIARDMPBX2WEIPRVHOQI4OEIPMJIZZRA, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.021462369 (81h49m44.892545414s) [noise=true , address=145.79.227.49:4359 , piece_id=FDOP6PX5XFB5OJVC7H4S6ZFY4FT5AAWOLIH43M4GTLCZS5YKIM5Q, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.021981778 (81h49m44.893064813s) [noise=true , address=145.79.227.52:4454 , piece_id=7YBPVY5EXGAIQKJEP5TCXX5KRFBUBC45QQI6CONCS5LGUMRU3Y4Q, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.022307669 (81h49m44.893390704s) [noise=true , address=145.79.227.9:4384 , piece_id=3JXASIXRKWJ573Q466LHQLNOZXF4UC2A6LRJKZE2KBOY2N444EUA, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.022633470 (81h49m44.893716505s) [noise=true , address=64.49.13.170:4722 , piece_id=Y5ZZFYYBICQCKHXREFINQ6LA6IUU4MAS4OA4SJXFBC2BFMIG4IMQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.023012681 (81h49m44.894095716s) [noise=true , address=77.90.31.16:4755 , piece_id=M2EBCSYLP54HIYVMV2WFXQRTLSISUJKZPGQRB6K6EEWSRKJKWVYA, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.023282415 (81h49m44.89436546s) [noise=true , address=145.79.227.4:4205 , piece_id=BWNYAT67IELSPLQTYMAM2O76OLQWUTXACQXR4VZSAQE67NYOASRA, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.023555585 (81h49m44.89463862s) [noise=true , address=77.90.31.16:4736 , piece_id=BHZSQ7MXD2X4YEGL6IYMUYCHURUXDRRRJDYPTBVZ44SORXMPSO5A, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.023817675 (81h49m44.89490071s) [noise=true , address=77.90.31.15:4713 , piece_id=BCW7S3OAKKGDYPH7RP7B6VJJSG5HNCR2DTLMHZDRLO4EEOPH7FKQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.024446281 (81h49m44.895529327s) [noise=true , address=77.90.31.17:4769 , piece_id=66IBD3RPUO7QFO4ALJSH5W4ELFKPLDOHSMIQ7UWNGGDV3OXZGUQQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.024703942 (81h49m44.895786987s) [noise=true , address=145.79.227.39:4037 , piece_id=PGBSSKDC7LGLKZE3FGX6FHAC6H45RMOLJFVQHCJPWWAVZSOSIG5A, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:46.437570978 (81h49m46.308654013s) [upload=71502]: Next returned nil segment Dec 9 01:18:46.437586438 (81h49m46.308669483s) [upload=71502]: Waiting for error group managing segments... Dec 9 01:18:46.796243098 (81h49m46.667326134s) [noise=true , address=64.49.13.170:4722 , piece_id=Y5ZZFYYBICQCKHXREFINQ6LA6IUU4MAS4OA4SJXFBC2BFMIG4IMQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.894539060 (81h49m46.765622105s) [noise=true , address=145.79.227.39:4037 , piece_id=PGBSSKDC7LGLKZE3FGX6FHAC6H45RMOLJFVQHCJPWWAVZSOSIG5A, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.906718262 (81h49m46.777801297s) [noise=true , address=145.79.227.9:4384 , piece_id=3JXASIXRKWJ573Q466LHQLNOZXF4UC2A6LRJKZE2KBOY2N444EUA, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.914490982 (81h49m46.785574027s) [noise=true , address=145.79.227.49:4359 , piece_id=FDOP6PX5XFB5OJVC7H4S6ZFY4FT5AAWOLIH43M4GTLCZS5YKIM5Q, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.928206178 (81h49m46.799289213s) [noise=true , address=145.79.227.52:4454 , piece_id=7YBPVY5EXGAIQKJEP5TCXX5KRFBUBC45QQI6CONCS5LGUMRU3Y4Q, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.953023966 (81h49m46.824107001s) [noise=true , address=145.79.227.12:4769 , piece_id=YTUOFFOTJBPKJUKCRDIY6S65ZP3W5XJCW7GU6GLSFHDUOPOTQO4Q, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.994863524 (81h49m46.865946569s) [noise=true , address=77.90.31.16:4755 , piece_id=M2EBCSYLP54HIYVMV2WFXQRTLSISUJKZPGQRB6K6EEWSRKJKWVYA, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.998843783 (81h49m46.869926818s) [noise=true , address=77.90.31.18:4830 , piece_id=3CTHRQHH7RNBOS6DDU3PUHE5TGEQMB4UBDSJEXE67GM7PXJIEO4A, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.002758567 (81h49m46.873841602s) [noise=true , address=77.90.31.17:4769 , piece_id=66IBD3RPUO7QFO4ALJSH5W4ELFKPLDOHSMIQ7UWNGGDV3OXZGUQQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.010756346 (81h49m46.881839382s) [noise=true , address=77.90.31.11:4570 , piece_id=HIPMDO4ST4XFQDT3M7PFIIARDMPBX2WEIPRVHOQI4OEIPMJIZZRA, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.088053593 (81h49m46.959136638s) [noise=true , address=45.91.116.10:4098 , piece_id=Q6MKNY37MMBVHUOMWJCTPE5BPXWG7G7RLEHQKLXYSDSI2KKYTPNQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.094063277 (81h49m46.965146322s) [noise=true , address=77.90.31.10:4530 , piece_id=XVCYKZOJ2H3U5RWMWW3BIBW37YK3ZQUHNIJHSGGTP7R2X244Y55Q, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.101953681 (81h49m46.973036726s) [noise=true , address=45.91.116.5:4306 , piece_id=5S4PWZULVPGCP4I4PSINBYKIOCBPLFIUOZE5QO4CKTQ3I7P76BQQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.109207003 (81h49m46.980290038s) [noise=true , address=77.90.31.10:4538 , piece_id=KTOSBWL7QVBHLWCAMOZNTHRCLWMUUHCLIPI2W52KOHNJ55NLNRWQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.109225769 (81h49m46.980308804s) [seg_pos={0 0}, upload=71502]: Segment reached optimal threshold of 14 pieces. Dec 9 01:18:47.109408547 (81h49m46.980491592s) [noise=true , address=77.90.31.15:4713 , piece_id=BCW7S3OAKKGDYPH7RP7B6VJJSG5HNCR2DTLMHZDRLO4EEOPH7FKQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Liv9RcEgfjxodPuadijzkcUGgJEy7LfKqHGFhaJCKbcxKkNN8): protocol: expected piece hash; context canceled; EOF Dec 9 01:18:47.109416322 (81h49m46.980499357s) [noise=true , address=77.90.31.15:4713 , piece_id=BCW7S3OAKKGDYPH7RP7B6VJJSG5HNCR2DTLMHZDRLO4EEOPH7FKQ, seg_pos={0 0}, upload=71502]: Long tail cancellation due to optimal threshold Dec 9 01:18:47.109443945 (81h49m46.98052698s) [noise=true , address=45.91.116.9:4287 , piece_id=GUUNM5MV3J7OELO633ESY3EMGLJOZKXNUKT3SRI7GKYJDYIC6SPA, seg_pos={0 0}, upload=71502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cNuooGmeprkK8fRRBY5ZSAbWda7fdfPbYwbg3fV2TKGsYikwf): protocol: expected piece hash; context canceled; EOF Dec 9 01:18:47.109452361 (81h49m46.980535396s) [noise=true , address=45.91.116.9:4287 , piece_id=GUUNM5MV3J7OELO633ESY3EMGLJOZKXNUKT3SRI7GKYJDYIC6SPA, seg_pos={0 0}, upload=71502]: Long tail cancellation due to optimal threshold Dec 9 01:18:47.109521943 (81h49m46.980604978s) [noise=true , address=145.79.227.4:4205 , piece_id=BWNYAT67IELSPLQTYMAM2O76OLQWUTXACQXR4VZSAQE67NYOASRA, seg_pos={0 0}, upload=71502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pih2cgjPo5YeP6B7g8diM7p11VRJfHy5H34GUT9CK77W2cTLr): protocol: expected piece hash; context canceled; EOF Dec 9 01:18:47.109527924 (81h49m46.980610969s) [noise=true , address=145.79.227.4:4205 , piece_id=BWNYAT67IELSPLQTYMAM2O76OLQWUTXACQXR4VZSAQE67NYOASRA, seg_pos={0 0}, upload=71502]: Long tail cancellation due to optimal threshold Dec 9 01:18:47.109629788 (81h49m46.980712823s) [noise=true , address=77.90.31.16:4736 , piece_id=BHZSQ7MXD2X4YEGL6IYMUYCHURUXDRRRJDYPTBVZ44SORXMPSO5A, seg_pos={0 0}, upload=71502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: expected piece hash; context canceled; EOF Dec 9 01:18:47.109635399 (81h49m46.980718444s) [noise=true , address=77.90.31.16:4736 , piece_id=BHZSQ7MXD2X4YEGL6IYMUYCHURUXDRRRJDYPTBVZ44SORXMPSO5A, seg_pos={0 0}, upload=71502]: Long tail cancellation due to optimal threshold Dec 9 01:18:47.109669504 (81h49m46.980752549s) [seg_pos={0 0}, upload=71502]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 9 01:18:47.109759546 (81h49m46.980842591s) [upload=71502]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 01:18:47.109766839 (81h49m46.980849874s) [upload=71502]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 01:19:58.768357796 (81h50m58.639440831s) [upload=71502]: Done uploading segments... Dec 9 01:19:58.768409825 (81h50m58.63949286s) [upload=71502]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.8149459s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [....X............X.X........................................] duration: 5.61331399s trace: https://jaeger.dev.storj.io/trace/4a1f040f80b86635 FAIL (2025-12-09 01:57:55 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.613351391s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 1.993877509s trace: https://jaeger.dev.storj.io/trace/36455c540a91ba90 FAIL (2025-12-09 01:57:38 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 1.993903017s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [......................................X.............X.......] duration: 3.406393174s trace: https://jaeger.dev.storj.io/trace/39c123c18fd52c67 FAIL (2025-12-09 01:57:27 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.406419193s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 2.00276914s trace: https://jaeger.dev.storj.io/trace/36455c540a91ba90 FAIL (2025-12-09 01:57:23 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 2.002797654s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 5.850395982s trace: https://jaeger.dev.storj.io/trace/1b9ac1acdb153d00 FAIL (2025-12-09 01:57:06 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== A, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.018287744 (81h49m44.88937079s) [noise=true , address=45.91.116.10:4098 , piece_id=Q6MKNY37MMBVHUOMWJCTPE5BPXWG7G7RLEHQKLXYSDSI2KKYTPNQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.018979372 (81h49m44.890062467s) [noise=true , address=45.91.116.5:4306 , piece_id=5S4PWZULVPGCP4I4PSINBYKIOCBPLFIUOZE5QO4CKTQ3I7P76BQQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.019582951 (81h49m44.890666006s) [noise=true , address=77.90.31.10:4538 , piece_id=KTOSBWL7QVBHLWCAMOZNTHRCLWMUUHCLIPI2W52KOHNJ55NLNRWQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.019819150 (81h49m44.890902196s) [noise=true , address=77.90.31.10:4530 , piece_id=XVCYKZOJ2H3U5RWMWW3BIBW37YK3ZQUHNIJHSGGTP7R2X244Y55Q, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.020245322 (81h49m44.891328377s) [noise=true , address=45.91.116.9:4287 , piece_id=GUUNM5MV3J7OELO633ESY3EMGLJOZKXNUKT3SRI7GKYJDYIC6SPA, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.020660623 (81h49m44.891743658s) [noise=true , address=145.79.227.12:4769 , piece_id=YTUOFFOTJBPKJUKCRDIY6S65ZP3W5XJCW7GU6GLSFHDUOPOTQO4Q, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.021074250 (81h49m44.892157295s) [noise=true , address=77.90.31.11:4570 , piece_id=HIPMDO4ST4XFQDT3M7PFIIARDMPBX2WEIPRVHOQI4OEIPMJIZZRA, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.021462369 (81h49m44.892545414s) [noise=true , address=145.79.227.49:4359 , piece_id=FDOP6PX5XFB5OJVC7H4S6ZFY4FT5AAWOLIH43M4GTLCZS5YKIM5Q, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.021981778 (81h49m44.893064813s) [noise=true , address=145.79.227.52:4454 , piece_id=7YBPVY5EXGAIQKJEP5TCXX5KRFBUBC45QQI6CONCS5LGUMRU3Y4Q, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.022307669 (81h49m44.893390704s) [noise=true , address=145.79.227.9:4384 , piece_id=3JXASIXRKWJ573Q466LHQLNOZXF4UC2A6LRJKZE2KBOY2N444EUA, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.022633470 (81h49m44.893716505s) [noise=true , address=64.49.13.170:4722 , piece_id=Y5ZZFYYBICQCKHXREFINQ6LA6IUU4MAS4OA4SJXFBC2BFMIG4IMQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.023012681 (81h49m44.894095716s) [noise=true , address=77.90.31.16:4755 , piece_id=M2EBCSYLP54HIYVMV2WFXQRTLSISUJKZPGQRB6K6EEWSRKJKWVYA, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.023282415 (81h49m44.89436546s) [noise=true , address=145.79.227.4:4205 , piece_id=BWNYAT67IELSPLQTYMAM2O76OLQWUTXACQXR4VZSAQE67NYOASRA, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.023555585 (81h49m44.89463862s) [noise=true , address=77.90.31.16:4736 , piece_id=BHZSQ7MXD2X4YEGL6IYMUYCHURUXDRRRJDYPTBVZ44SORXMPSO5A, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.023817675 (81h49m44.89490071s) [noise=true , address=77.90.31.15:4713 , piece_id=BCW7S3OAKKGDYPH7RP7B6VJJSG5HNCR2DTLMHZDRLO4EEOPH7FKQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.024446281 (81h49m44.895529327s) [noise=true , address=77.90.31.17:4769 , piece_id=66IBD3RPUO7QFO4ALJSH5W4ELFKPLDOHSMIQ7UWNGGDV3OXZGUQQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.024703942 (81h49m44.895786987s) [noise=true , address=145.79.227.39:4037 , piece_id=PGBSSKDC7LGLKZE3FGX6FHAC6H45RMOLJFVQHCJPWWAVZSOSIG5A, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:46.437570978 (81h49m46.308654013s) [upload=71502]: Next returned nil segment Dec 9 01:18:46.437586438 (81h49m46.308669483s) [upload=71502]: Waiting for error group managing segments... Dec 9 01:18:46.796243098 (81h49m46.667326134s) [noise=true , address=64.49.13.170:4722 , piece_id=Y5ZZFYYBICQCKHXREFINQ6LA6IUU4MAS4OA4SJXFBC2BFMIG4IMQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.894539060 (81h49m46.765622105s) [noise=true , address=145.79.227.39:4037 , piece_id=PGBSSKDC7LGLKZE3FGX6FHAC6H45RMOLJFVQHCJPWWAVZSOSIG5A, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.906718262 (81h49m46.777801297s) [noise=true , address=145.79.227.9:4384 , piece_id=3JXASIXRKWJ573Q466LHQLNOZXF4UC2A6LRJKZE2KBOY2N444EUA, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.914490982 (81h49m46.785574027s) [noise=true , address=145.79.227.49:4359 , piece_id=FDOP6PX5XFB5OJVC7H4S6ZFY4FT5AAWOLIH43M4GTLCZS5YKIM5Q, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.928206178 (81h49m46.799289213s) [noise=true , address=145.79.227.52:4454 , piece_id=7YBPVY5EXGAIQKJEP5TCXX5KRFBUBC45QQI6CONCS5LGUMRU3Y4Q, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.953023966 (81h49m46.824107001s) [noise=true , address=145.79.227.12:4769 , piece_id=YTUOFFOTJBPKJUKCRDIY6S65ZP3W5XJCW7GU6GLSFHDUOPOTQO4Q, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.994863524 (81h49m46.865946569s) [noise=true , address=77.90.31.16:4755 , piece_id=M2EBCSYLP54HIYVMV2WFXQRTLSISUJKZPGQRB6K6EEWSRKJKWVYA, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.998843783 (81h49m46.869926818s) [noise=true , address=77.90.31.18:4830 , piece_id=3CTHRQHH7RNBOS6DDU3PUHE5TGEQMB4UBDSJEXE67GM7PXJIEO4A, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.002758567 (81h49m46.873841602s) [noise=true , address=77.90.31.17:4769 , piece_id=66IBD3RPUO7QFO4ALJSH5W4ELFKPLDOHSMIQ7UWNGGDV3OXZGUQQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.010756346 (81h49m46.881839382s) [noise=true , address=77.90.31.11:4570 , piece_id=HIPMDO4ST4XFQDT3M7PFIIARDMPBX2WEIPRVHOQI4OEIPMJIZZRA, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.088053593 (81h49m46.959136638s) [noise=true , address=45.91.116.10:4098 , piece_id=Q6MKNY37MMBVHUOMWJCTPE5BPXWG7G7RLEHQKLXYSDSI2KKYTPNQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.094063277 (81h49m46.965146322s) [noise=true , address=77.90.31.10:4530 , piece_id=XVCYKZOJ2H3U5RWMWW3BIBW37YK3ZQUHNIJHSGGTP7R2X244Y55Q, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.101953681 (81h49m46.973036726s) [noise=true , address=45.91.116.5:4306 , piece_id=5S4PWZULVPGCP4I4PSINBYKIOCBPLFIUOZE5QO4CKTQ3I7P76BQQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.109207003 (81h49m46.980290038s) [noise=true , address=77.90.31.10:4538 , piece_id=KTOSBWL7QVBHLWCAMOZNTHRCLWMUUHCLIPI2W52KOHNJ55NLNRWQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.109225769 (81h49m46.980308804s) [seg_pos={0 0}, upload=71502]: Segment reached optimal threshold of 14 pieces. Dec 9 01:18:47.109408547 (81h49m46.980491592s) [noise=true , address=77.90.31.15:4713 , piece_id=BCW7S3OAKKGDYPH7RP7B6VJJSG5HNCR2DTLMHZDRLO4EEOPH7FKQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Liv9RcEgfjxodPuadijzkcUGgJEy7LfKqHGFhaJCKbcxKkNN8): protocol: expected piece hash; context canceled; EOF Dec 9 01:18:47.109416322 (81h49m46.980499357s) [noise=true , address=77.90.31.15:4713 , piece_id=BCW7S3OAKKGDYPH7RP7B6VJJSG5HNCR2DTLMHZDRLO4EEOPH7FKQ, seg_pos={0 0}, upload=71502]: Long tail cancellation due to optimal threshold Dec 9 01:18:47.109443945 (81h49m46.98052698s) [noise=true , address=45.91.116.9:4287 , piece_id=GUUNM5MV3J7OELO633ESY3EMGLJOZKXNUKT3SRI7GKYJDYIC6SPA, seg_pos={0 0}, upload=71502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cNuooGmeprkK8fRRBY5ZSAbWda7fdfPbYwbg3fV2TKGsYikwf): protocol: expected piece hash; context canceled; EOF Dec 9 01:18:47.109452361 (81h49m46.980535396s) [noise=true , address=45.91.116.9:4287 , piece_id=GUUNM5MV3J7OELO633ESY3EMGLJOZKXNUKT3SRI7GKYJDYIC6SPA, seg_pos={0 0}, upload=71502]: Long tail cancellation due to optimal threshold Dec 9 01:18:47.109521943 (81h49m46.980604978s) [noise=true , address=145.79.227.4:4205 , piece_id=BWNYAT67IELSPLQTYMAM2O76OLQWUTXACQXR4VZSAQE67NYOASRA, seg_pos={0 0}, upload=71502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pih2cgjPo5YeP6B7g8diM7p11VRJfHy5H34GUT9CK77W2cTLr): protocol: expected piece hash; context canceled; EOF Dec 9 01:18:47.109527924 (81h49m46.980610969s) [noise=true , address=145.79.227.4:4205 , piece_id=BWNYAT67IELSPLQTYMAM2O76OLQWUTXACQXR4VZSAQE67NYOASRA, seg_pos={0 0}, upload=71502]: Long tail cancellation due to optimal threshold Dec 9 01:18:47.109629788 (81h49m46.980712823s) [noise=true , address=77.90.31.16:4736 , piece_id=BHZSQ7MXD2X4YEGL6IYMUYCHURUXDRRRJDYPTBVZ44SORXMPSO5A, seg_pos={0 0}, upload=71502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: expected piece hash; context canceled; EOF Dec 9 01:18:47.109635399 (81h49m46.980718444s) [noise=true , address=77.90.31.16:4736 , piece_id=BHZSQ7MXD2X4YEGL6IYMUYCHURUXDRRRJDYPTBVZ44SORXMPSO5A, seg_pos={0 0}, upload=71502]: Long tail cancellation due to optimal threshold Dec 9 01:18:47.109669504 (81h49m46.980752549s) [seg_pos={0 0}, upload=71502]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 9 01:18:47.109759546 (81h49m46.980842591s) [upload=71502]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 01:18:47.109766839 (81h49m46.980849874s) [upload=71502]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 01:19:58.768357796 (81h50m58.639440831s) [upload=71502]: Done uploading segments... Dec 9 01:19:58.768409825 (81h50m58.63949286s) [upload=71502]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.85042141s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [.....X............X.X.......................................] duration: 5.906231533s trace: https://jaeger.dev.storj.io/trace/4a1f040f80b86635 FAIL (2025-12-09 01:57:05 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.906278492s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 1.986636681s trace: https://jaeger.dev.storj.io/trace/36455c540a91ba90 FAIL (2025-12-09 01:56:39 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 1.986673181s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 1.95872283s trace: https://jaeger.dev.storj.io/trace/36455c540a91ba90 FAIL (2025-12-09 01:56:17 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 1.95875457s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.......................................X.............X......] duration: 3.530996869s trace: https://jaeger.dev.storj.io/trace/39c123c18fd52c67 FAIL (2025-12-09 01:56:17 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.531021717s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 5.776839573s trace: https://jaeger.dev.storj.io/trace/1b9ac1acdb153d00 FAIL (2025-12-09 01:55:59 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== A, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.018287744 (81h49m44.88937079s) [noise=true , address=45.91.116.10:4098 , piece_id=Q6MKNY37MMBVHUOMWJCTPE5BPXWG7G7RLEHQKLXYSDSI2KKYTPNQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.018979372 (81h49m44.890062467s) [noise=true , address=45.91.116.5:4306 , piece_id=5S4PWZULVPGCP4I4PSINBYKIOCBPLFIUOZE5QO4CKTQ3I7P76BQQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.019582951 (81h49m44.890666006s) [noise=true , address=77.90.31.10:4538 , piece_id=KTOSBWL7QVBHLWCAMOZNTHRCLWMUUHCLIPI2W52KOHNJ55NLNRWQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.019819150 (81h49m44.890902196s) [noise=true , address=77.90.31.10:4530 , piece_id=XVCYKZOJ2H3U5RWMWW3BIBW37YK3ZQUHNIJHSGGTP7R2X244Y55Q, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.020245322 (81h49m44.891328377s) [noise=true , address=45.91.116.9:4287 , piece_id=GUUNM5MV3J7OELO633ESY3EMGLJOZKXNUKT3SRI7GKYJDYIC6SPA, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.020660623 (81h49m44.891743658s) [noise=true , address=145.79.227.12:4769 , piece_id=YTUOFFOTJBPKJUKCRDIY6S65ZP3W5XJCW7GU6GLSFHDUOPOTQO4Q, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.021074250 (81h49m44.892157295s) [noise=true , address=77.90.31.11:4570 , piece_id=HIPMDO4ST4XFQDT3M7PFIIARDMPBX2WEIPRVHOQI4OEIPMJIZZRA, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.021462369 (81h49m44.892545414s) [noise=true , address=145.79.227.49:4359 , piece_id=FDOP6PX5XFB5OJVC7H4S6ZFY4FT5AAWOLIH43M4GTLCZS5YKIM5Q, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.021981778 (81h49m44.893064813s) [noise=true , address=145.79.227.52:4454 , piece_id=7YBPVY5EXGAIQKJEP5TCXX5KRFBUBC45QQI6CONCS5LGUMRU3Y4Q, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.022307669 (81h49m44.893390704s) [noise=true , address=145.79.227.9:4384 , piece_id=3JXASIXRKWJ573Q466LHQLNOZXF4UC2A6LRJKZE2KBOY2N444EUA, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.022633470 (81h49m44.893716505s) [noise=true , address=64.49.13.170:4722 , piece_id=Y5ZZFYYBICQCKHXREFINQ6LA6IUU4MAS4OA4SJXFBC2BFMIG4IMQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.023012681 (81h49m44.894095716s) [noise=true , address=77.90.31.16:4755 , piece_id=M2EBCSYLP54HIYVMV2WFXQRTLSISUJKZPGQRB6K6EEWSRKJKWVYA, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.023282415 (81h49m44.89436546s) [noise=true , address=145.79.227.4:4205 , piece_id=BWNYAT67IELSPLQTYMAM2O76OLQWUTXACQXR4VZSAQE67NYOASRA, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.023555585 (81h49m44.89463862s) [noise=true , address=77.90.31.16:4736 , piece_id=BHZSQ7MXD2X4YEGL6IYMUYCHURUXDRRRJDYPTBVZ44SORXMPSO5A, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.023817675 (81h49m44.89490071s) [noise=true , address=77.90.31.15:4713 , piece_id=BCW7S3OAKKGDYPH7RP7B6VJJSG5HNCR2DTLMHZDRLO4EEOPH7FKQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.024446281 (81h49m44.895529327s) [noise=true , address=77.90.31.17:4769 , piece_id=66IBD3RPUO7QFO4ALJSH5W4ELFKPLDOHSMIQ7UWNGGDV3OXZGUQQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.024703942 (81h49m44.895786987s) [noise=true , address=145.79.227.39:4037 , piece_id=PGBSSKDC7LGLKZE3FGX6FHAC6H45RMOLJFVQHCJPWWAVZSOSIG5A, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:46.437570978 (81h49m46.308654013s) [upload=71502]: Next returned nil segment Dec 9 01:18:46.437586438 (81h49m46.308669483s) [upload=71502]: Waiting for error group managing segments... Dec 9 01:18:46.796243098 (81h49m46.667326134s) [noise=true , address=64.49.13.170:4722 , piece_id=Y5ZZFYYBICQCKHXREFINQ6LA6IUU4MAS4OA4SJXFBC2BFMIG4IMQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.894539060 (81h49m46.765622105s) [noise=true , address=145.79.227.39:4037 , piece_id=PGBSSKDC7LGLKZE3FGX6FHAC6H45RMOLJFVQHCJPWWAVZSOSIG5A, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.906718262 (81h49m46.777801297s) [noise=true , address=145.79.227.9:4384 , piece_id=3JXASIXRKWJ573Q466LHQLNOZXF4UC2A6LRJKZE2KBOY2N444EUA, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.914490982 (81h49m46.785574027s) [noise=true , address=145.79.227.49:4359 , piece_id=FDOP6PX5XFB5OJVC7H4S6ZFY4FT5AAWOLIH43M4GTLCZS5YKIM5Q, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.928206178 (81h49m46.799289213s) [noise=true , address=145.79.227.52:4454 , piece_id=7YBPVY5EXGAIQKJEP5TCXX5KRFBUBC45QQI6CONCS5LGUMRU3Y4Q, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.953023966 (81h49m46.824107001s) [noise=true , address=145.79.227.12:4769 , piece_id=YTUOFFOTJBPKJUKCRDIY6S65ZP3W5XJCW7GU6GLSFHDUOPOTQO4Q, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.994863524 (81h49m46.865946569s) [noise=true , address=77.90.31.16:4755 , piece_id=M2EBCSYLP54HIYVMV2WFXQRTLSISUJKZPGQRB6K6EEWSRKJKWVYA, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.998843783 (81h49m46.869926818s) [noise=true , address=77.90.31.18:4830 , piece_id=3CTHRQHH7RNBOS6DDU3PUHE5TGEQMB4UBDSJEXE67GM7PXJIEO4A, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.002758567 (81h49m46.873841602s) [noise=true , address=77.90.31.17:4769 , piece_id=66IBD3RPUO7QFO4ALJSH5W4ELFKPLDOHSMIQ7UWNGGDV3OXZGUQQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.010756346 (81h49m46.881839382s) [noise=true , address=77.90.31.11:4570 , piece_id=HIPMDO4ST4XFQDT3M7PFIIARDMPBX2WEIPRVHOQI4OEIPMJIZZRA, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.088053593 (81h49m46.959136638s) [noise=true , address=45.91.116.10:4098 , piece_id=Q6MKNY37MMBVHUOMWJCTPE5BPXWG7G7RLEHQKLXYSDSI2KKYTPNQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.094063277 (81h49m46.965146322s) [noise=true , address=77.90.31.10:4530 , piece_id=XVCYKZOJ2H3U5RWMWW3BIBW37YK3ZQUHNIJHSGGTP7R2X244Y55Q, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.101953681 (81h49m46.973036726s) [noise=true , address=45.91.116.5:4306 , piece_id=5S4PWZULVPGCP4I4PSINBYKIOCBPLFIUOZE5QO4CKTQ3I7P76BQQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.109207003 (81h49m46.980290038s) [noise=true , address=77.90.31.10:4538 , piece_id=KTOSBWL7QVBHLWCAMOZNTHRCLWMUUHCLIPI2W52KOHNJ55NLNRWQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.109225769 (81h49m46.980308804s) [seg_pos={0 0}, upload=71502]: Segment reached optimal threshold of 14 pieces. Dec 9 01:18:47.109408547 (81h49m46.980491592s) [noise=true , address=77.90.31.15:4713 , piece_id=BCW7S3OAKKGDYPH7RP7B6VJJSG5HNCR2DTLMHZDRLO4EEOPH7FKQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Liv9RcEgfjxodPuadijzkcUGgJEy7LfKqHGFhaJCKbcxKkNN8): protocol: expected piece hash; context canceled; EOF Dec 9 01:18:47.109416322 (81h49m46.980499357s) [noise=true , address=77.90.31.15:4713 , piece_id=BCW7S3OAKKGDYPH7RP7B6VJJSG5HNCR2DTLMHZDRLO4EEOPH7FKQ, seg_pos={0 0}, upload=71502]: Long tail cancellation due to optimal threshold Dec 9 01:18:47.109443945 (81h49m46.98052698s) [noise=true , address=45.91.116.9:4287 , piece_id=GUUNM5MV3J7OELO633ESY3EMGLJOZKXNUKT3SRI7GKYJDYIC6SPA, seg_pos={0 0}, upload=71502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cNuooGmeprkK8fRRBY5ZSAbWda7fdfPbYwbg3fV2TKGsYikwf): protocol: expected piece hash; context canceled; EOF Dec 9 01:18:47.109452361 (81h49m46.980535396s) [noise=true , address=45.91.116.9:4287 , piece_id=GUUNM5MV3J7OELO633ESY3EMGLJOZKXNUKT3SRI7GKYJDYIC6SPA, seg_pos={0 0}, upload=71502]: Long tail cancellation due to optimal threshold Dec 9 01:18:47.109521943 (81h49m46.980604978s) [noise=true , address=145.79.227.4:4205 , piece_id=BWNYAT67IELSPLQTYMAM2O76OLQWUTXACQXR4VZSAQE67NYOASRA, seg_pos={0 0}, upload=71502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pih2cgjPo5YeP6B7g8diM7p11VRJfHy5H34GUT9CK77W2cTLr): protocol: expected piece hash; context canceled; EOF Dec 9 01:18:47.109527924 (81h49m46.980610969s) [noise=true , address=145.79.227.4:4205 , piece_id=BWNYAT67IELSPLQTYMAM2O76OLQWUTXACQXR4VZSAQE67NYOASRA, seg_pos={0 0}, upload=71502]: Long tail cancellation due to optimal threshold Dec 9 01:18:47.109629788 (81h49m46.980712823s) [noise=true , address=77.90.31.16:4736 , piece_id=BHZSQ7MXD2X4YEGL6IYMUYCHURUXDRRRJDYPTBVZ44SORXMPSO5A, seg_pos={0 0}, upload=71502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: expected piece hash; context canceled; EOF Dec 9 01:18:47.109635399 (81h49m46.980718444s) [noise=true , address=77.90.31.16:4736 , piece_id=BHZSQ7MXD2X4YEGL6IYMUYCHURUXDRRRJDYPTBVZ44SORXMPSO5A, seg_pos={0 0}, upload=71502]: Long tail cancellation due to optimal threshold Dec 9 01:18:47.109669504 (81h49m46.980752549s) [seg_pos={0 0}, upload=71502]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 9 01:18:47.109759546 (81h49m46.980842591s) [upload=71502]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 01:18:47.109766839 (81h49m46.980849874s) [upload=71502]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 01:19:58.768357796 (81h50m58.639440831s) [upload=71502]: Done uploading segments... Dec 9 01:19:58.768409825 (81h50m58.63949286s) [upload=71502]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.776889157s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [......X............X.X......................................] duration: 5.894598167s trace: https://jaeger.dev.storj.io/trace/4a1f040f80b86635 FAIL (2025-12-09 01:55:57 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.894639956s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [........................................X.............X.....] duration: 3.362362279s trace: https://jaeger.dev.storj.io/trace/39c123c18fd52c67 FAIL (2025-12-09 01:55:48 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.362387788s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.........................................X.............X....] duration: 3.540251732s trace: https://jaeger.dev.storj.io/trace/39c123c18fd52c67 FAIL (2025-12-09 01:55:26 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.54039254s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 2.01637051s trace: https://jaeger.dev.storj.io/trace/36455c540a91ba90 FAIL (2025-12-09 01:55:23 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 2.016396951s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 5.965923781s trace: https://jaeger.dev.storj.io/trace/1b9ac1acdb153d00 FAIL (2025-12-09 01:55:02 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== A, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.018287744 (81h49m44.88937079s) [noise=true , address=45.91.116.10:4098 , piece_id=Q6MKNY37MMBVHUOMWJCTPE5BPXWG7G7RLEHQKLXYSDSI2KKYTPNQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.018979372 (81h49m44.890062467s) [noise=true , address=45.91.116.5:4306 , piece_id=5S4PWZULVPGCP4I4PSINBYKIOCBPLFIUOZE5QO4CKTQ3I7P76BQQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.019582951 (81h49m44.890666006s) [noise=true , address=77.90.31.10:4538 , piece_id=KTOSBWL7QVBHLWCAMOZNTHRCLWMUUHCLIPI2W52KOHNJ55NLNRWQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.019819150 (81h49m44.890902196s) [noise=true , address=77.90.31.10:4530 , piece_id=XVCYKZOJ2H3U5RWMWW3BIBW37YK3ZQUHNIJHSGGTP7R2X244Y55Q, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.020245322 (81h49m44.891328377s) [noise=true , address=45.91.116.9:4287 , piece_id=GUUNM5MV3J7OELO633ESY3EMGLJOZKXNUKT3SRI7GKYJDYIC6SPA, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.020660623 (81h49m44.891743658s) [noise=true , address=145.79.227.12:4769 , piece_id=YTUOFFOTJBPKJUKCRDIY6S65ZP3W5XJCW7GU6GLSFHDUOPOTQO4Q, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.021074250 (81h49m44.892157295s) [noise=true , address=77.90.31.11:4570 , piece_id=HIPMDO4ST4XFQDT3M7PFIIARDMPBX2WEIPRVHOQI4OEIPMJIZZRA, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.021462369 (81h49m44.892545414s) [noise=true , address=145.79.227.49:4359 , piece_id=FDOP6PX5XFB5OJVC7H4S6ZFY4FT5AAWOLIH43M4GTLCZS5YKIM5Q, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.021981778 (81h49m44.893064813s) [noise=true , address=145.79.227.52:4454 , piece_id=7YBPVY5EXGAIQKJEP5TCXX5KRFBUBC45QQI6CONCS5LGUMRU3Y4Q, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.022307669 (81h49m44.893390704s) [noise=true , address=145.79.227.9:4384 , piece_id=3JXASIXRKWJ573Q466LHQLNOZXF4UC2A6LRJKZE2KBOY2N444EUA, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.022633470 (81h49m44.893716505s) [noise=true , address=64.49.13.170:4722 , piece_id=Y5ZZFYYBICQCKHXREFINQ6LA6IUU4MAS4OA4SJXFBC2BFMIG4IMQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.023012681 (81h49m44.894095716s) [noise=true , address=77.90.31.16:4755 , piece_id=M2EBCSYLP54HIYVMV2WFXQRTLSISUJKZPGQRB6K6EEWSRKJKWVYA, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.023282415 (81h49m44.89436546s) [noise=true , address=145.79.227.4:4205 , piece_id=BWNYAT67IELSPLQTYMAM2O76OLQWUTXACQXR4VZSAQE67NYOASRA, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.023555585 (81h49m44.89463862s) [noise=true , address=77.90.31.16:4736 , piece_id=BHZSQ7MXD2X4YEGL6IYMUYCHURUXDRRRJDYPTBVZ44SORXMPSO5A, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.023817675 (81h49m44.89490071s) [noise=true , address=77.90.31.15:4713 , piece_id=BCW7S3OAKKGDYPH7RP7B6VJJSG5HNCR2DTLMHZDRLO4EEOPH7FKQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.024446281 (81h49m44.895529327s) [noise=true , address=77.90.31.17:4769 , piece_id=66IBD3RPUO7QFO4ALJSH5W4ELFKPLDOHSMIQ7UWNGGDV3OXZGUQQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.024703942 (81h49m44.895786987s) [noise=true , address=145.79.227.39:4037 , piece_id=PGBSSKDC7LGLKZE3FGX6FHAC6H45RMOLJFVQHCJPWWAVZSOSIG5A, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:46.437570978 (81h49m46.308654013s) [upload=71502]: Next returned nil segment Dec 9 01:18:46.437586438 (81h49m46.308669483s) [upload=71502]: Waiting for error group managing segments... Dec 9 01:18:46.796243098 (81h49m46.667326134s) [noise=true , address=64.49.13.170:4722 , piece_id=Y5ZZFYYBICQCKHXREFINQ6LA6IUU4MAS4OA4SJXFBC2BFMIG4IMQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.894539060 (81h49m46.765622105s) [noise=true , address=145.79.227.39:4037 , piece_id=PGBSSKDC7LGLKZE3FGX6FHAC6H45RMOLJFVQHCJPWWAVZSOSIG5A, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.906718262 (81h49m46.777801297s) [noise=true , address=145.79.227.9:4384 , piece_id=3JXASIXRKWJ573Q466LHQLNOZXF4UC2A6LRJKZE2KBOY2N444EUA, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.914490982 (81h49m46.785574027s) [noise=true , address=145.79.227.49:4359 , piece_id=FDOP6PX5XFB5OJVC7H4S6ZFY4FT5AAWOLIH43M4GTLCZS5YKIM5Q, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.928206178 (81h49m46.799289213s) [noise=true , address=145.79.227.52:4454 , piece_id=7YBPVY5EXGAIQKJEP5TCXX5KRFBUBC45QQI6CONCS5LGUMRU3Y4Q, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.953023966 (81h49m46.824107001s) [noise=true , address=145.79.227.12:4769 , piece_id=YTUOFFOTJBPKJUKCRDIY6S65ZP3W5XJCW7GU6GLSFHDUOPOTQO4Q, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.994863524 (81h49m46.865946569s) [noise=true , address=77.90.31.16:4755 , piece_id=M2EBCSYLP54HIYVMV2WFXQRTLSISUJKZPGQRB6K6EEWSRKJKWVYA, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.998843783 (81h49m46.869926818s) [noise=true , address=77.90.31.18:4830 , piece_id=3CTHRQHH7RNBOS6DDU3PUHE5TGEQMB4UBDSJEXE67GM7PXJIEO4A, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.002758567 (81h49m46.873841602s) [noise=true , address=77.90.31.17:4769 , piece_id=66IBD3RPUO7QFO4ALJSH5W4ELFKPLDOHSMIQ7UWNGGDV3OXZGUQQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.010756346 (81h49m46.881839382s) [noise=true , address=77.90.31.11:4570 , piece_id=HIPMDO4ST4XFQDT3M7PFIIARDMPBX2WEIPRVHOQI4OEIPMJIZZRA, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.088053593 (81h49m46.959136638s) [noise=true , address=45.91.116.10:4098 , piece_id=Q6MKNY37MMBVHUOMWJCTPE5BPXWG7G7RLEHQKLXYSDSI2KKYTPNQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.094063277 (81h49m46.965146322s) [noise=true , address=77.90.31.10:4530 , piece_id=XVCYKZOJ2H3U5RWMWW3BIBW37YK3ZQUHNIJHSGGTP7R2X244Y55Q, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.101953681 (81h49m46.973036726s) [noise=true , address=45.91.116.5:4306 , piece_id=5S4PWZULVPGCP4I4PSINBYKIOCBPLFIUOZE5QO4CKTQ3I7P76BQQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.109207003 (81h49m46.980290038s) [noise=true , address=77.90.31.10:4538 , piece_id=KTOSBWL7QVBHLWCAMOZNTHRCLWMUUHCLIPI2W52KOHNJ55NLNRWQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.109225769 (81h49m46.980308804s) [seg_pos={0 0}, upload=71502]: Segment reached optimal threshold of 14 pieces. Dec 9 01:18:47.109408547 (81h49m46.980491592s) [noise=true , address=77.90.31.15:4713 , piece_id=BCW7S3OAKKGDYPH7RP7B6VJJSG5HNCR2DTLMHZDRLO4EEOPH7FKQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Liv9RcEgfjxodPuadijzkcUGgJEy7LfKqHGFhaJCKbcxKkNN8): protocol: expected piece hash; context canceled; EOF Dec 9 01:18:47.109416322 (81h49m46.980499357s) [noise=true , address=77.90.31.15:4713 , piece_id=BCW7S3OAKKGDYPH7RP7B6VJJSG5HNCR2DTLMHZDRLO4EEOPH7FKQ, seg_pos={0 0}, upload=71502]: Long tail cancellation due to optimal threshold Dec 9 01:18:47.109443945 (81h49m46.98052698s) [noise=true , address=45.91.116.9:4287 , piece_id=GUUNM5MV3J7OELO633ESY3EMGLJOZKXNUKT3SRI7GKYJDYIC6SPA, seg_pos={0 0}, upload=71502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cNuooGmeprkK8fRRBY5ZSAbWda7fdfPbYwbg3fV2TKGsYikwf): protocol: expected piece hash; context canceled; EOF Dec 9 01:18:47.109452361 (81h49m46.980535396s) [noise=true , address=45.91.116.9:4287 , piece_id=GUUNM5MV3J7OELO633ESY3EMGLJOZKXNUKT3SRI7GKYJDYIC6SPA, seg_pos={0 0}, upload=71502]: Long tail cancellation due to optimal threshold Dec 9 01:18:47.109521943 (81h49m46.980604978s) [noise=true , address=145.79.227.4:4205 , piece_id=BWNYAT67IELSPLQTYMAM2O76OLQWUTXACQXR4VZSAQE67NYOASRA, seg_pos={0 0}, upload=71502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pih2cgjPo5YeP6B7g8diM7p11VRJfHy5H34GUT9CK77W2cTLr): protocol: expected piece hash; context canceled; EOF Dec 9 01:18:47.109527924 (81h49m46.980610969s) [noise=true , address=145.79.227.4:4205 , piece_id=BWNYAT67IELSPLQTYMAM2O76OLQWUTXACQXR4VZSAQE67NYOASRA, seg_pos={0 0}, upload=71502]: Long tail cancellation due to optimal threshold Dec 9 01:18:47.109629788 (81h49m46.980712823s) [noise=true , address=77.90.31.16:4736 , piece_id=BHZSQ7MXD2X4YEGL6IYMUYCHURUXDRRRJDYPTBVZ44SORXMPSO5A, seg_pos={0 0}, upload=71502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: expected piece hash; context canceled; EOF Dec 9 01:18:47.109635399 (81h49m46.980718444s) [noise=true , address=77.90.31.16:4736 , piece_id=BHZSQ7MXD2X4YEGL6IYMUYCHURUXDRRRJDYPTBVZ44SORXMPSO5A, seg_pos={0 0}, upload=71502]: Long tail cancellation due to optimal threshold Dec 9 01:18:47.109669504 (81h49m46.980752549s) [seg_pos={0 0}, upload=71502]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 9 01:18:47.109759546 (81h49m46.980842591s) [upload=71502]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 01:18:47.109766839 (81h49m46.980849874s) [upload=71502]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 01:19:58.768357796 (81h50m58.639440831s) [upload=71502]: Done uploading segments... Dec 9 01:19:58.768409825 (81h50m58.63949286s) [upload=71502]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.966160342s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [.......X............X.X.....................................] duration: 5.740099634s trace: https://jaeger.dev.storj.io/trace/4a1f040f80b86635 FAIL (2025-12-09 01:54:54 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.740216247s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [..........................................X.............X...] duration: 3.579092561s trace: https://jaeger.dev.storj.io/trace/39c123c18fd52c67 FAIL (2025-12-09 01:54:22 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.579116346s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 3.388897095s trace: https://jaeger.dev.storj.io/trace/36455c540a91ba90 FAIL (2025-12-09 01:54:19 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 3.3889309s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [........X............X.X....................................] duration: 5.916341498s trace: https://jaeger.dev.storj.io/trace/4a1f040f80b86635 FAIL (2025-12-09 01:54:03 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.916370222s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 5.857857064s trace: https://jaeger.dev.storj.io/trace/1b9ac1acdb153d00 FAIL (2025-12-09 01:53:56 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== A, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.018287744 (81h49m44.88937079s) [noise=true , address=45.91.116.10:4098 , piece_id=Q6MKNY37MMBVHUOMWJCTPE5BPXWG7G7RLEHQKLXYSDSI2KKYTPNQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.018979372 (81h49m44.890062467s) [noise=true , address=45.91.116.5:4306 , piece_id=5S4PWZULVPGCP4I4PSINBYKIOCBPLFIUOZE5QO4CKTQ3I7P76BQQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.019582951 (81h49m44.890666006s) [noise=true , address=77.90.31.10:4538 , piece_id=KTOSBWL7QVBHLWCAMOZNTHRCLWMUUHCLIPI2W52KOHNJ55NLNRWQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.019819150 (81h49m44.890902196s) [noise=true , address=77.90.31.10:4530 , piece_id=XVCYKZOJ2H3U5RWMWW3BIBW37YK3ZQUHNIJHSGGTP7R2X244Y55Q, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.020245322 (81h49m44.891328377s) [noise=true , address=45.91.116.9:4287 , piece_id=GUUNM5MV3J7OELO633ESY3EMGLJOZKXNUKT3SRI7GKYJDYIC6SPA, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.020660623 (81h49m44.891743658s) [noise=true , address=145.79.227.12:4769 , piece_id=YTUOFFOTJBPKJUKCRDIY6S65ZP3W5XJCW7GU6GLSFHDUOPOTQO4Q, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.021074250 (81h49m44.892157295s) [noise=true , address=77.90.31.11:4570 , piece_id=HIPMDO4ST4XFQDT3M7PFIIARDMPBX2WEIPRVHOQI4OEIPMJIZZRA, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.021462369 (81h49m44.892545414s) [noise=true , address=145.79.227.49:4359 , piece_id=FDOP6PX5XFB5OJVC7H4S6ZFY4FT5AAWOLIH43M4GTLCZS5YKIM5Q, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.021981778 (81h49m44.893064813s) [noise=true , address=145.79.227.52:4454 , piece_id=7YBPVY5EXGAIQKJEP5TCXX5KRFBUBC45QQI6CONCS5LGUMRU3Y4Q, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.022307669 (81h49m44.893390704s) [noise=true , address=145.79.227.9:4384 , piece_id=3JXASIXRKWJ573Q466LHQLNOZXF4UC2A6LRJKZE2KBOY2N444EUA, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.022633470 (81h49m44.893716505s) [noise=true , address=64.49.13.170:4722 , piece_id=Y5ZZFYYBICQCKHXREFINQ6LA6IUU4MAS4OA4SJXFBC2BFMIG4IMQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.023012681 (81h49m44.894095716s) [noise=true , address=77.90.31.16:4755 , piece_id=M2EBCSYLP54HIYVMV2WFXQRTLSISUJKZPGQRB6K6EEWSRKJKWVYA, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.023282415 (81h49m44.89436546s) [noise=true , address=145.79.227.4:4205 , piece_id=BWNYAT67IELSPLQTYMAM2O76OLQWUTXACQXR4VZSAQE67NYOASRA, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.023555585 (81h49m44.89463862s) [noise=true , address=77.90.31.16:4736 , piece_id=BHZSQ7MXD2X4YEGL6IYMUYCHURUXDRRRJDYPTBVZ44SORXMPSO5A, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.023817675 (81h49m44.89490071s) [noise=true , address=77.90.31.15:4713 , piece_id=BCW7S3OAKKGDYPH7RP7B6VJJSG5HNCR2DTLMHZDRLO4EEOPH7FKQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.024446281 (81h49m44.895529327s) [noise=true , address=77.90.31.17:4769 , piece_id=66IBD3RPUO7QFO4ALJSH5W4ELFKPLDOHSMIQ7UWNGGDV3OXZGUQQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.024703942 (81h49m44.895786987s) [noise=true , address=145.79.227.39:4037 , piece_id=PGBSSKDC7LGLKZE3FGX6FHAC6H45RMOLJFVQHCJPWWAVZSOSIG5A, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:46.437570978 (81h49m46.308654013s) [upload=71502]: Next returned nil segment Dec 9 01:18:46.437586438 (81h49m46.308669483s) [upload=71502]: Waiting for error group managing segments... Dec 9 01:18:46.796243098 (81h49m46.667326134s) [noise=true , address=64.49.13.170:4722 , piece_id=Y5ZZFYYBICQCKHXREFINQ6LA6IUU4MAS4OA4SJXFBC2BFMIG4IMQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.894539060 (81h49m46.765622105s) [noise=true , address=145.79.227.39:4037 , piece_id=PGBSSKDC7LGLKZE3FGX6FHAC6H45RMOLJFVQHCJPWWAVZSOSIG5A, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.906718262 (81h49m46.777801297s) [noise=true , address=145.79.227.9:4384 , piece_id=3JXASIXRKWJ573Q466LHQLNOZXF4UC2A6LRJKZE2KBOY2N444EUA, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.914490982 (81h49m46.785574027s) [noise=true , address=145.79.227.49:4359 , piece_id=FDOP6PX5XFB5OJVC7H4S6ZFY4FT5AAWOLIH43M4GTLCZS5YKIM5Q, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.928206178 (81h49m46.799289213s) [noise=true , address=145.79.227.52:4454 , piece_id=7YBPVY5EXGAIQKJEP5TCXX5KRFBUBC45QQI6CONCS5LGUMRU3Y4Q, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.953023966 (81h49m46.824107001s) [noise=true , address=145.79.227.12:4769 , piece_id=YTUOFFOTJBPKJUKCRDIY6S65ZP3W5XJCW7GU6GLSFHDUOPOTQO4Q, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.994863524 (81h49m46.865946569s) [noise=true , address=77.90.31.16:4755 , piece_id=M2EBCSYLP54HIYVMV2WFXQRTLSISUJKZPGQRB6K6EEWSRKJKWVYA, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.998843783 (81h49m46.869926818s) [noise=true , address=77.90.31.18:4830 , piece_id=3CTHRQHH7RNBOS6DDU3PUHE5TGEQMB4UBDSJEXE67GM7PXJIEO4A, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.002758567 (81h49m46.873841602s) [noise=true , address=77.90.31.17:4769 , piece_id=66IBD3RPUO7QFO4ALJSH5W4ELFKPLDOHSMIQ7UWNGGDV3OXZGUQQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.010756346 (81h49m46.881839382s) [noise=true , address=77.90.31.11:4570 , piece_id=HIPMDO4ST4XFQDT3M7PFIIARDMPBX2WEIPRVHOQI4OEIPMJIZZRA, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.088053593 (81h49m46.959136638s) [noise=true , address=45.91.116.10:4098 , piece_id=Q6MKNY37MMBVHUOMWJCTPE5BPXWG7G7RLEHQKLXYSDSI2KKYTPNQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.094063277 (81h49m46.965146322s) [noise=true , address=77.90.31.10:4530 , piece_id=XVCYKZOJ2H3U5RWMWW3BIBW37YK3ZQUHNIJHSGGTP7R2X244Y55Q, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.101953681 (81h49m46.973036726s) [noise=true , address=45.91.116.5:4306 , piece_id=5S4PWZULVPGCP4I4PSINBYKIOCBPLFIUOZE5QO4CKTQ3I7P76BQQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.109207003 (81h49m46.980290038s) [noise=true , address=77.90.31.10:4538 , piece_id=KTOSBWL7QVBHLWCAMOZNTHRCLWMUUHCLIPI2W52KOHNJ55NLNRWQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.109225769 (81h49m46.980308804s) [seg_pos={0 0}, upload=71502]: Segment reached optimal threshold of 14 pieces. Dec 9 01:18:47.109408547 (81h49m46.980491592s) [noise=true , address=77.90.31.15:4713 , piece_id=BCW7S3OAKKGDYPH7RP7B6VJJSG5HNCR2DTLMHZDRLO4EEOPH7FKQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Liv9RcEgfjxodPuadijzkcUGgJEy7LfKqHGFhaJCKbcxKkNN8): protocol: expected piece hash; context canceled; EOF Dec 9 01:18:47.109416322 (81h49m46.980499357s) [noise=true , address=77.90.31.15:4713 , piece_id=BCW7S3OAKKGDYPH7RP7B6VJJSG5HNCR2DTLMHZDRLO4EEOPH7FKQ, seg_pos={0 0}, upload=71502]: Long tail cancellation due to optimal threshold Dec 9 01:18:47.109443945 (81h49m46.98052698s) [noise=true , address=45.91.116.9:4287 , piece_id=GUUNM5MV3J7OELO633ESY3EMGLJOZKXNUKT3SRI7GKYJDYIC6SPA, seg_pos={0 0}, upload=71502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cNuooGmeprkK8fRRBY5ZSAbWda7fdfPbYwbg3fV2TKGsYikwf): protocol: expected piece hash; context canceled; EOF Dec 9 01:18:47.109452361 (81h49m46.980535396s) [noise=true , address=45.91.116.9:4287 , piece_id=GUUNM5MV3J7OELO633ESY3EMGLJOZKXNUKT3SRI7GKYJDYIC6SPA, seg_pos={0 0}, upload=71502]: Long tail cancellation due to optimal threshold Dec 9 01:18:47.109521943 (81h49m46.980604978s) [noise=true , address=145.79.227.4:4205 , piece_id=BWNYAT67IELSPLQTYMAM2O76OLQWUTXACQXR4VZSAQE67NYOASRA, seg_pos={0 0}, upload=71502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pih2cgjPo5YeP6B7g8diM7p11VRJfHy5H34GUT9CK77W2cTLr): protocol: expected piece hash; context canceled; EOF Dec 9 01:18:47.109527924 (81h49m46.980610969s) [noise=true , address=145.79.227.4:4205 , piece_id=BWNYAT67IELSPLQTYMAM2O76OLQWUTXACQXR4VZSAQE67NYOASRA, seg_pos={0 0}, upload=71502]: Long tail cancellation due to optimal threshold Dec 9 01:18:47.109629788 (81h49m46.980712823s) [noise=true , address=77.90.31.16:4736 , piece_id=BHZSQ7MXD2X4YEGL6IYMUYCHURUXDRRRJDYPTBVZ44SORXMPSO5A, seg_pos={0 0}, upload=71502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: expected piece hash; context canceled; EOF Dec 9 01:18:47.109635399 (81h49m46.980718444s) [noise=true , address=77.90.31.16:4736 , piece_id=BHZSQ7MXD2X4YEGL6IYMUYCHURUXDRRRJDYPTBVZ44SORXMPSO5A, seg_pos={0 0}, upload=71502]: Long tail cancellation due to optimal threshold Dec 9 01:18:47.109669504 (81h49m46.980752549s) [seg_pos={0 0}, upload=71502]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 9 01:18:47.109759546 (81h49m46.980842591s) [upload=71502]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 01:18:47.109766839 (81h49m46.980849874s) [upload=71502]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 01:19:58.768357796 (81h50m58.639440831s) [upload=71502]: Done uploading segments... Dec 9 01:19:58.768409825 (81h50m58.63949286s) [upload=71502]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.857872272s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 1.976706864s trace: https://jaeger.dev.storj.io/trace/36455c540a91ba90 FAIL (2025-12-09 01:53:16 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 1.976734638s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [...........................................X.............X..] duration: 3.440302933s trace: https://jaeger.dev.storj.io/trace/39c123c18fd52c67 FAIL (2025-12-09 01:53:16 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.440331146s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 5.692115434s trace: https://jaeger.dev.storj.io/trace/1b9ac1acdb153d00 FAIL (2025-12-09 01:53:14 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== A, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.018287744 (81h49m44.88937079s) [noise=true , address=45.91.116.10:4098 , piece_id=Q6MKNY37MMBVHUOMWJCTPE5BPXWG7G7RLEHQKLXYSDSI2KKYTPNQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.018979372 (81h49m44.890062467s) [noise=true , address=45.91.116.5:4306 , piece_id=5S4PWZULVPGCP4I4PSINBYKIOCBPLFIUOZE5QO4CKTQ3I7P76BQQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.019582951 (81h49m44.890666006s) [noise=true , address=77.90.31.10:4538 , piece_id=KTOSBWL7QVBHLWCAMOZNTHRCLWMUUHCLIPI2W52KOHNJ55NLNRWQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.019819150 (81h49m44.890902196s) [noise=true , address=77.90.31.10:4530 , piece_id=XVCYKZOJ2H3U5RWMWW3BIBW37YK3ZQUHNIJHSGGTP7R2X244Y55Q, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.020245322 (81h49m44.891328377s) [noise=true , address=45.91.116.9:4287 , piece_id=GUUNM5MV3J7OELO633ESY3EMGLJOZKXNUKT3SRI7GKYJDYIC6SPA, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.020660623 (81h49m44.891743658s) [noise=true , address=145.79.227.12:4769 , piece_id=YTUOFFOTJBPKJUKCRDIY6S65ZP3W5XJCW7GU6GLSFHDUOPOTQO4Q, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.021074250 (81h49m44.892157295s) [noise=true , address=77.90.31.11:4570 , piece_id=HIPMDO4ST4XFQDT3M7PFIIARDMPBX2WEIPRVHOQI4OEIPMJIZZRA, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.021462369 (81h49m44.892545414s) [noise=true , address=145.79.227.49:4359 , piece_id=FDOP6PX5XFB5OJVC7H4S6ZFY4FT5AAWOLIH43M4GTLCZS5YKIM5Q, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.021981778 (81h49m44.893064813s) [noise=true , address=145.79.227.52:4454 , piece_id=7YBPVY5EXGAIQKJEP5TCXX5KRFBUBC45QQI6CONCS5LGUMRU3Y4Q, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.022307669 (81h49m44.893390704s) [noise=true , address=145.79.227.9:4384 , piece_id=3JXASIXRKWJ573Q466LHQLNOZXF4UC2A6LRJKZE2KBOY2N444EUA, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.022633470 (81h49m44.893716505s) [noise=true , address=64.49.13.170:4722 , piece_id=Y5ZZFYYBICQCKHXREFINQ6LA6IUU4MAS4OA4SJXFBC2BFMIG4IMQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.023012681 (81h49m44.894095716s) [noise=true , address=77.90.31.16:4755 , piece_id=M2EBCSYLP54HIYVMV2WFXQRTLSISUJKZPGQRB6K6EEWSRKJKWVYA, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.023282415 (81h49m44.89436546s) [noise=true , address=145.79.227.4:4205 , piece_id=BWNYAT67IELSPLQTYMAM2O76OLQWUTXACQXR4VZSAQE67NYOASRA, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.023555585 (81h49m44.89463862s) [noise=true , address=77.90.31.16:4736 , piece_id=BHZSQ7MXD2X4YEGL6IYMUYCHURUXDRRRJDYPTBVZ44SORXMPSO5A, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.023817675 (81h49m44.89490071s) [noise=true , address=77.90.31.15:4713 , piece_id=BCW7S3OAKKGDYPH7RP7B6VJJSG5HNCR2DTLMHZDRLO4EEOPH7FKQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.024446281 (81h49m44.895529327s) [noise=true , address=77.90.31.17:4769 , piece_id=66IBD3RPUO7QFO4ALJSH5W4ELFKPLDOHSMIQ7UWNGGDV3OXZGUQQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.024703942 (81h49m44.895786987s) [noise=true , address=145.79.227.39:4037 , piece_id=PGBSSKDC7LGLKZE3FGX6FHAC6H45RMOLJFVQHCJPWWAVZSOSIG5A, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:46.437570978 (81h49m46.308654013s) [upload=71502]: Next returned nil segment Dec 9 01:18:46.437586438 (81h49m46.308669483s) [upload=71502]: Waiting for error group managing segments... Dec 9 01:18:46.796243098 (81h49m46.667326134s) [noise=true , address=64.49.13.170:4722 , piece_id=Y5ZZFYYBICQCKHXREFINQ6LA6IUU4MAS4OA4SJXFBC2BFMIG4IMQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.894539060 (81h49m46.765622105s) [noise=true , address=145.79.227.39:4037 , piece_id=PGBSSKDC7LGLKZE3FGX6FHAC6H45RMOLJFVQHCJPWWAVZSOSIG5A, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.906718262 (81h49m46.777801297s) [noise=true , address=145.79.227.9:4384 , piece_id=3JXASIXRKWJ573Q466LHQLNOZXF4UC2A6LRJKZE2KBOY2N444EUA, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.914490982 (81h49m46.785574027s) [noise=true , address=145.79.227.49:4359 , piece_id=FDOP6PX5XFB5OJVC7H4S6ZFY4FT5AAWOLIH43M4GTLCZS5YKIM5Q, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.928206178 (81h49m46.799289213s) [noise=true , address=145.79.227.52:4454 , piece_id=7YBPVY5EXGAIQKJEP5TCXX5KRFBUBC45QQI6CONCS5LGUMRU3Y4Q, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.953023966 (81h49m46.824107001s) [noise=true , address=145.79.227.12:4769 , piece_id=YTUOFFOTJBPKJUKCRDIY6S65ZP3W5XJCW7GU6GLSFHDUOPOTQO4Q, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.994863524 (81h49m46.865946569s) [noise=true , address=77.90.31.16:4755 , piece_id=M2EBCSYLP54HIYVMV2WFXQRTLSISUJKZPGQRB6K6EEWSRKJKWVYA, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.998843783 (81h49m46.869926818s) [noise=true , address=77.90.31.18:4830 , piece_id=3CTHRQHH7RNBOS6DDU3PUHE5TGEQMB4UBDSJEXE67GM7PXJIEO4A, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.002758567 (81h49m46.873841602s) [noise=true , address=77.90.31.17:4769 , piece_id=66IBD3RPUO7QFO4ALJSH5W4ELFKPLDOHSMIQ7UWNGGDV3OXZGUQQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.010756346 (81h49m46.881839382s) [noise=true , address=77.90.31.11:4570 , piece_id=HIPMDO4ST4XFQDT3M7PFIIARDMPBX2WEIPRVHOQI4OEIPMJIZZRA, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.088053593 (81h49m46.959136638s) [noise=true , address=45.91.116.10:4098 , piece_id=Q6MKNY37MMBVHUOMWJCTPE5BPXWG7G7RLEHQKLXYSDSI2KKYTPNQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.094063277 (81h49m46.965146322s) [noise=true , address=77.90.31.10:4530 , piece_id=XVCYKZOJ2H3U5RWMWW3BIBW37YK3ZQUHNIJHSGGTP7R2X244Y55Q, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.101953681 (81h49m46.973036726s) [noise=true , address=45.91.116.5:4306 , piece_id=5S4PWZULVPGCP4I4PSINBYKIOCBPLFIUOZE5QO4CKTQ3I7P76BQQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.109207003 (81h49m46.980290038s) [noise=true , address=77.90.31.10:4538 , piece_id=KTOSBWL7QVBHLWCAMOZNTHRCLWMUUHCLIPI2W52KOHNJ55NLNRWQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.109225769 (81h49m46.980308804s) [seg_pos={0 0}, upload=71502]: Segment reached optimal threshold of 14 pieces. Dec 9 01:18:47.109408547 (81h49m46.980491592s) [noise=true , address=77.90.31.15:4713 , piece_id=BCW7S3OAKKGDYPH7RP7B6VJJSG5HNCR2DTLMHZDRLO4EEOPH7FKQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Liv9RcEgfjxodPuadijzkcUGgJEy7LfKqHGFhaJCKbcxKkNN8): protocol: expected piece hash; context canceled; EOF Dec 9 01:18:47.109416322 (81h49m46.980499357s) [noise=true , address=77.90.31.15:4713 , piece_id=BCW7S3OAKKGDYPH7RP7B6VJJSG5HNCR2DTLMHZDRLO4EEOPH7FKQ, seg_pos={0 0}, upload=71502]: Long tail cancellation due to optimal threshold Dec 9 01:18:47.109443945 (81h49m46.98052698s) [noise=true , address=45.91.116.9:4287 , piece_id=GUUNM5MV3J7OELO633ESY3EMGLJOZKXNUKT3SRI7GKYJDYIC6SPA, seg_pos={0 0}, upload=71502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cNuooGmeprkK8fRRBY5ZSAbWda7fdfPbYwbg3fV2TKGsYikwf): protocol: expected piece hash; context canceled; EOF Dec 9 01:18:47.109452361 (81h49m46.980535396s) [noise=true , address=45.91.116.9:4287 , piece_id=GUUNM5MV3J7OELO633ESY3EMGLJOZKXNUKT3SRI7GKYJDYIC6SPA, seg_pos={0 0}, upload=71502]: Long tail cancellation due to optimal threshold Dec 9 01:18:47.109521943 (81h49m46.980604978s) [noise=true , address=145.79.227.4:4205 , piece_id=BWNYAT67IELSPLQTYMAM2O76OLQWUTXACQXR4VZSAQE67NYOASRA, seg_pos={0 0}, upload=71502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pih2cgjPo5YeP6B7g8diM7p11VRJfHy5H34GUT9CK77W2cTLr): protocol: expected piece hash; context canceled; EOF Dec 9 01:18:47.109527924 (81h49m46.980610969s) [noise=true , address=145.79.227.4:4205 , piece_id=BWNYAT67IELSPLQTYMAM2O76OLQWUTXACQXR4VZSAQE67NYOASRA, seg_pos={0 0}, upload=71502]: Long tail cancellation due to optimal threshold Dec 9 01:18:47.109629788 (81h49m46.980712823s) [noise=true , address=77.90.31.16:4736 , piece_id=BHZSQ7MXD2X4YEGL6IYMUYCHURUXDRRRJDYPTBVZ44SORXMPSO5A, seg_pos={0 0}, upload=71502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: expected piece hash; context canceled; EOF Dec 9 01:18:47.109635399 (81h49m46.980718444s) [noise=true , address=77.90.31.16:4736 , piece_id=BHZSQ7MXD2X4YEGL6IYMUYCHURUXDRRRJDYPTBVZ44SORXMPSO5A, seg_pos={0 0}, upload=71502]: Long tail cancellation due to optimal threshold Dec 9 01:18:47.109669504 (81h49m46.980752549s) [seg_pos={0 0}, upload=71502]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 9 01:18:47.109759546 (81h49m46.980842591s) [upload=71502]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 01:18:47.109766839 (81h49m46.980849874s) [upload=71502]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 01:19:58.768357796 (81h50m58.639440831s) [upload=71502]: Done uploading segments... Dec 9 01:19:58.768409825 (81h50m58.63949286s) [upload=71502]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.692144871s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [X........X............X.X...................................] duration: 5.670655281s trace: https://jaeger.dev.storj.io/trace/4a1f040f80b86635 FAIL (2025-12-09 01:53:06 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.670831185s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 5.753697617s trace: https://jaeger.dev.storj.io/trace/1b9ac1acdb153d00 FAIL (2025-12-09 01:52:46 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== A, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.018287744 (81h49m44.88937079s) [noise=true , address=45.91.116.10:4098 , piece_id=Q6MKNY37MMBVHUOMWJCTPE5BPXWG7G7RLEHQKLXYSDSI2KKYTPNQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.018979372 (81h49m44.890062467s) [noise=true , address=45.91.116.5:4306 , piece_id=5S4PWZULVPGCP4I4PSINBYKIOCBPLFIUOZE5QO4CKTQ3I7P76BQQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.019582951 (81h49m44.890666006s) [noise=true , address=77.90.31.10:4538 , piece_id=KTOSBWL7QVBHLWCAMOZNTHRCLWMUUHCLIPI2W52KOHNJ55NLNRWQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.019819150 (81h49m44.890902196s) [noise=true , address=77.90.31.10:4530 , piece_id=XVCYKZOJ2H3U5RWMWW3BIBW37YK3ZQUHNIJHSGGTP7R2X244Y55Q, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.020245322 (81h49m44.891328377s) [noise=true , address=45.91.116.9:4287 , piece_id=GUUNM5MV3J7OELO633ESY3EMGLJOZKXNUKT3SRI7GKYJDYIC6SPA, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.020660623 (81h49m44.891743658s) [noise=true , address=145.79.227.12:4769 , piece_id=YTUOFFOTJBPKJUKCRDIY6S65ZP3W5XJCW7GU6GLSFHDUOPOTQO4Q, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.021074250 (81h49m44.892157295s) [noise=true , address=77.90.31.11:4570 , piece_id=HIPMDO4ST4XFQDT3M7PFIIARDMPBX2WEIPRVHOQI4OEIPMJIZZRA, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.021462369 (81h49m44.892545414s) [noise=true , address=145.79.227.49:4359 , piece_id=FDOP6PX5XFB5OJVC7H4S6ZFY4FT5AAWOLIH43M4GTLCZS5YKIM5Q, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.021981778 (81h49m44.893064813s) [noise=true , address=145.79.227.52:4454 , piece_id=7YBPVY5EXGAIQKJEP5TCXX5KRFBUBC45QQI6CONCS5LGUMRU3Y4Q, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.022307669 (81h49m44.893390704s) [noise=true , address=145.79.227.9:4384 , piece_id=3JXASIXRKWJ573Q466LHQLNOZXF4UC2A6LRJKZE2KBOY2N444EUA, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.022633470 (81h49m44.893716505s) [noise=true , address=64.49.13.170:4722 , piece_id=Y5ZZFYYBICQCKHXREFINQ6LA6IUU4MAS4OA4SJXFBC2BFMIG4IMQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.023012681 (81h49m44.894095716s) [noise=true , address=77.90.31.16:4755 , piece_id=M2EBCSYLP54HIYVMV2WFXQRTLSISUJKZPGQRB6K6EEWSRKJKWVYA, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.023282415 (81h49m44.89436546s) [noise=true , address=145.79.227.4:4205 , piece_id=BWNYAT67IELSPLQTYMAM2O76OLQWUTXACQXR4VZSAQE67NYOASRA, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.023555585 (81h49m44.89463862s) [noise=true , address=77.90.31.16:4736 , piece_id=BHZSQ7MXD2X4YEGL6IYMUYCHURUXDRRRJDYPTBVZ44SORXMPSO5A, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.023817675 (81h49m44.89490071s) [noise=true , address=77.90.31.15:4713 , piece_id=BCW7S3OAKKGDYPH7RP7B6VJJSG5HNCR2DTLMHZDRLO4EEOPH7FKQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.024446281 (81h49m44.895529327s) [noise=true , address=77.90.31.17:4769 , piece_id=66IBD3RPUO7QFO4ALJSH5W4ELFKPLDOHSMIQ7UWNGGDV3OXZGUQQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.024703942 (81h49m44.895786987s) [noise=true , address=145.79.227.39:4037 , piece_id=PGBSSKDC7LGLKZE3FGX6FHAC6H45RMOLJFVQHCJPWWAVZSOSIG5A, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:46.437570978 (81h49m46.308654013s) [upload=71502]: Next returned nil segment Dec 9 01:18:46.437586438 (81h49m46.308669483s) [upload=71502]: Waiting for error group managing segments... Dec 9 01:18:46.796243098 (81h49m46.667326134s) [noise=true , address=64.49.13.170:4722 , piece_id=Y5ZZFYYBICQCKHXREFINQ6LA6IUU4MAS4OA4SJXFBC2BFMIG4IMQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.894539060 (81h49m46.765622105s) [noise=true , address=145.79.227.39:4037 , piece_id=PGBSSKDC7LGLKZE3FGX6FHAC6H45RMOLJFVQHCJPWWAVZSOSIG5A, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.906718262 (81h49m46.777801297s) [noise=true , address=145.79.227.9:4384 , piece_id=3JXASIXRKWJ573Q466LHQLNOZXF4UC2A6LRJKZE2KBOY2N444EUA, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.914490982 (81h49m46.785574027s) [noise=true , address=145.79.227.49:4359 , piece_id=FDOP6PX5XFB5OJVC7H4S6ZFY4FT5AAWOLIH43M4GTLCZS5YKIM5Q, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.928206178 (81h49m46.799289213s) [noise=true , address=145.79.227.52:4454 , piece_id=7YBPVY5EXGAIQKJEP5TCXX5KRFBUBC45QQI6CONCS5LGUMRU3Y4Q, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.953023966 (81h49m46.824107001s) [noise=true , address=145.79.227.12:4769 , piece_id=YTUOFFOTJBPKJUKCRDIY6S65ZP3W5XJCW7GU6GLSFHDUOPOTQO4Q, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.994863524 (81h49m46.865946569s) [noise=true , address=77.90.31.16:4755 , piece_id=M2EBCSYLP54HIYVMV2WFXQRTLSISUJKZPGQRB6K6EEWSRKJKWVYA, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.998843783 (81h49m46.869926818s) [noise=true , address=77.90.31.18:4830 , piece_id=3CTHRQHH7RNBOS6DDU3PUHE5TGEQMB4UBDSJEXE67GM7PXJIEO4A, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.002758567 (81h49m46.873841602s) [noise=true , address=77.90.31.17:4769 , piece_id=66IBD3RPUO7QFO4ALJSH5W4ELFKPLDOHSMIQ7UWNGGDV3OXZGUQQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.010756346 (81h49m46.881839382s) [noise=true , address=77.90.31.11:4570 , piece_id=HIPMDO4ST4XFQDT3M7PFIIARDMPBX2WEIPRVHOQI4OEIPMJIZZRA, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.088053593 (81h49m46.959136638s) [noise=true , address=45.91.116.10:4098 , piece_id=Q6MKNY37MMBVHUOMWJCTPE5BPXWG7G7RLEHQKLXYSDSI2KKYTPNQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.094063277 (81h49m46.965146322s) [noise=true , address=77.90.31.10:4530 , piece_id=XVCYKZOJ2H3U5RWMWW3BIBW37YK3ZQUHNIJHSGGTP7R2X244Y55Q, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.101953681 (81h49m46.973036726s) [noise=true , address=45.91.116.5:4306 , piece_id=5S4PWZULVPGCP4I4PSINBYKIOCBPLFIUOZE5QO4CKTQ3I7P76BQQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.109207003 (81h49m46.980290038s) [noise=true , address=77.90.31.10:4538 , piece_id=KTOSBWL7QVBHLWCAMOZNTHRCLWMUUHCLIPI2W52KOHNJ55NLNRWQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.109225769 (81h49m46.980308804s) [seg_pos={0 0}, upload=71502]: Segment reached optimal threshold of 14 pieces. Dec 9 01:18:47.109408547 (81h49m46.980491592s) [noise=true , address=77.90.31.15:4713 , piece_id=BCW7S3OAKKGDYPH7RP7B6VJJSG5HNCR2DTLMHZDRLO4EEOPH7FKQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Liv9RcEgfjxodPuadijzkcUGgJEy7LfKqHGFhaJCKbcxKkNN8): protocol: expected piece hash; context canceled; EOF Dec 9 01:18:47.109416322 (81h49m46.980499357s) [noise=true , address=77.90.31.15:4713 , piece_id=BCW7S3OAKKGDYPH7RP7B6VJJSG5HNCR2DTLMHZDRLO4EEOPH7FKQ, seg_pos={0 0}, upload=71502]: Long tail cancellation due to optimal threshold Dec 9 01:18:47.109443945 (81h49m46.98052698s) [noise=true , address=45.91.116.9:4287 , piece_id=GUUNM5MV3J7OELO633ESY3EMGLJOZKXNUKT3SRI7GKYJDYIC6SPA, seg_pos={0 0}, upload=71502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cNuooGmeprkK8fRRBY5ZSAbWda7fdfPbYwbg3fV2TKGsYikwf): protocol: expected piece hash; context canceled; EOF Dec 9 01:18:47.109452361 (81h49m46.980535396s) [noise=true , address=45.91.116.9:4287 , piece_id=GUUNM5MV3J7OELO633ESY3EMGLJOZKXNUKT3SRI7GKYJDYIC6SPA, seg_pos={0 0}, upload=71502]: Long tail cancellation due to optimal threshold Dec 9 01:18:47.109521943 (81h49m46.980604978s) [noise=true , address=145.79.227.4:4205 , piece_id=BWNYAT67IELSPLQTYMAM2O76OLQWUTXACQXR4VZSAQE67NYOASRA, seg_pos={0 0}, upload=71502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pih2cgjPo5YeP6B7g8diM7p11VRJfHy5H34GUT9CK77W2cTLr): protocol: expected piece hash; context canceled; EOF Dec 9 01:18:47.109527924 (81h49m46.980610969s) [noise=true , address=145.79.227.4:4205 , piece_id=BWNYAT67IELSPLQTYMAM2O76OLQWUTXACQXR4VZSAQE67NYOASRA, seg_pos={0 0}, upload=71502]: Long tail cancellation due to optimal threshold Dec 9 01:18:47.109629788 (81h49m46.980712823s) [noise=true , address=77.90.31.16:4736 , piece_id=BHZSQ7MXD2X4YEGL6IYMUYCHURUXDRRRJDYPTBVZ44SORXMPSO5A, seg_pos={0 0}, upload=71502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: expected piece hash; context canceled; EOF Dec 9 01:18:47.109635399 (81h49m46.980718444s) [noise=true , address=77.90.31.16:4736 , piece_id=BHZSQ7MXD2X4YEGL6IYMUYCHURUXDRRRJDYPTBVZ44SORXMPSO5A, seg_pos={0 0}, upload=71502]: Long tail cancellation due to optimal threshold Dec 9 01:18:47.109669504 (81h49m46.980752549s) [seg_pos={0 0}, upload=71502]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 9 01:18:47.109759546 (81h49m46.980842591s) [upload=71502]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 01:18:47.109766839 (81h49m46.980849874s) [upload=71502]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 01:19:58.768357796 (81h50m58.639440831s) [upload=71502]: Done uploading segments... Dec 9 01:19:58.768409825 (81h50m58.63949286s) [upload=71502]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.753723567s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [............................................X.............X.] duration: 3.414917785s trace: https://jaeger.dev.storj.io/trace/39c123c18fd52c67 FAIL (2025-12-09 01:52:41 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.414945408s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 1.977704564s trace: https://jaeger.dev.storj.io/trace/36455c540a91ba90 FAIL (2025-12-09 01:52:22 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 1.977730686s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [.X........X............X.X..................................] duration: 5.871165006s trace: https://jaeger.dev.storj.io/trace/4a1f040f80b86635 FAIL (2025-12-09 01:52:22 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.871190033s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.............................................X.............X] duration: 5.001323129s trace: https://jaeger.dev.storj.io/trace/39c123c18fd52c67 FAIL (2025-12-09 01:52:14 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001347145s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [..X........X............X.X.................................] duration: 5.827758149s trace: https://jaeger.dev.storj.io/trace/4a1f040f80b86635 FAIL (2025-12-09 01:52:02 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.82781092s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 2.001408614s trace: https://jaeger.dev.storj.io/trace/36455c540a91ba90 FAIL (2025-12-09 01:51:50 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 2.001439912s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 5.944347183s trace: https://jaeger.dev.storj.io/trace/1b9ac1acdb153d00 FAIL (2025-12-09 01:51:47 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== A, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.018287744 (81h49m44.88937079s) [noise=true , address=45.91.116.10:4098 , piece_id=Q6MKNY37MMBVHUOMWJCTPE5BPXWG7G7RLEHQKLXYSDSI2KKYTPNQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.018979372 (81h49m44.890062467s) [noise=true , address=45.91.116.5:4306 , piece_id=5S4PWZULVPGCP4I4PSINBYKIOCBPLFIUOZE5QO4CKTQ3I7P76BQQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.019582951 (81h49m44.890666006s) [noise=true , address=77.90.31.10:4538 , piece_id=KTOSBWL7QVBHLWCAMOZNTHRCLWMUUHCLIPI2W52KOHNJ55NLNRWQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.019819150 (81h49m44.890902196s) [noise=true , address=77.90.31.10:4530 , piece_id=XVCYKZOJ2H3U5RWMWW3BIBW37YK3ZQUHNIJHSGGTP7R2X244Y55Q, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.020245322 (81h49m44.891328377s) [noise=true , address=45.91.116.9:4287 , piece_id=GUUNM5MV3J7OELO633ESY3EMGLJOZKXNUKT3SRI7GKYJDYIC6SPA, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.020660623 (81h49m44.891743658s) [noise=true , address=145.79.227.12:4769 , piece_id=YTUOFFOTJBPKJUKCRDIY6S65ZP3W5XJCW7GU6GLSFHDUOPOTQO4Q, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.021074250 (81h49m44.892157295s) [noise=true , address=77.90.31.11:4570 , piece_id=HIPMDO4ST4XFQDT3M7PFIIARDMPBX2WEIPRVHOQI4OEIPMJIZZRA, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.021462369 (81h49m44.892545414s) [noise=true , address=145.79.227.49:4359 , piece_id=FDOP6PX5XFB5OJVC7H4S6ZFY4FT5AAWOLIH43M4GTLCZS5YKIM5Q, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.021981778 (81h49m44.893064813s) [noise=true , address=145.79.227.52:4454 , piece_id=7YBPVY5EXGAIQKJEP5TCXX5KRFBUBC45QQI6CONCS5LGUMRU3Y4Q, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.022307669 (81h49m44.893390704s) [noise=true , address=145.79.227.9:4384 , piece_id=3JXASIXRKWJ573Q466LHQLNOZXF4UC2A6LRJKZE2KBOY2N444EUA, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.022633470 (81h49m44.893716505s) [noise=true , address=64.49.13.170:4722 , piece_id=Y5ZZFYYBICQCKHXREFINQ6LA6IUU4MAS4OA4SJXFBC2BFMIG4IMQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.023012681 (81h49m44.894095716s) [noise=true , address=77.90.31.16:4755 , piece_id=M2EBCSYLP54HIYVMV2WFXQRTLSISUJKZPGQRB6K6EEWSRKJKWVYA, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.023282415 (81h49m44.89436546s) [noise=true , address=145.79.227.4:4205 , piece_id=BWNYAT67IELSPLQTYMAM2O76OLQWUTXACQXR4VZSAQE67NYOASRA, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.023555585 (81h49m44.89463862s) [noise=true , address=77.90.31.16:4736 , piece_id=BHZSQ7MXD2X4YEGL6IYMUYCHURUXDRRRJDYPTBVZ44SORXMPSO5A, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.023817675 (81h49m44.89490071s) [noise=true , address=77.90.31.15:4713 , piece_id=BCW7S3OAKKGDYPH7RP7B6VJJSG5HNCR2DTLMHZDRLO4EEOPH7FKQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.024446281 (81h49m44.895529327s) [noise=true , address=77.90.31.17:4769 , piece_id=66IBD3RPUO7QFO4ALJSH5W4ELFKPLDOHSMIQ7UWNGGDV3OXZGUQQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.024703942 (81h49m44.895786987s) [noise=true , address=145.79.227.39:4037 , piece_id=PGBSSKDC7LGLKZE3FGX6FHAC6H45RMOLJFVQHCJPWWAVZSOSIG5A, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:46.437570978 (81h49m46.308654013s) [upload=71502]: Next returned nil segment Dec 9 01:18:46.437586438 (81h49m46.308669483s) [upload=71502]: Waiting for error group managing segments... Dec 9 01:18:46.796243098 (81h49m46.667326134s) [noise=true , address=64.49.13.170:4722 , piece_id=Y5ZZFYYBICQCKHXREFINQ6LA6IUU4MAS4OA4SJXFBC2BFMIG4IMQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.894539060 (81h49m46.765622105s) [noise=true , address=145.79.227.39:4037 , piece_id=PGBSSKDC7LGLKZE3FGX6FHAC6H45RMOLJFVQHCJPWWAVZSOSIG5A, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.906718262 (81h49m46.777801297s) [noise=true , address=145.79.227.9:4384 , piece_id=3JXASIXRKWJ573Q466LHQLNOZXF4UC2A6LRJKZE2KBOY2N444EUA, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.914490982 (81h49m46.785574027s) [noise=true , address=145.79.227.49:4359 , piece_id=FDOP6PX5XFB5OJVC7H4S6ZFY4FT5AAWOLIH43M4GTLCZS5YKIM5Q, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.928206178 (81h49m46.799289213s) [noise=true , address=145.79.227.52:4454 , piece_id=7YBPVY5EXGAIQKJEP5TCXX5KRFBUBC45QQI6CONCS5LGUMRU3Y4Q, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.953023966 (81h49m46.824107001s) [noise=true , address=145.79.227.12:4769 , piece_id=YTUOFFOTJBPKJUKCRDIY6S65ZP3W5XJCW7GU6GLSFHDUOPOTQO4Q, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.994863524 (81h49m46.865946569s) [noise=true , address=77.90.31.16:4755 , piece_id=M2EBCSYLP54HIYVMV2WFXQRTLSISUJKZPGQRB6K6EEWSRKJKWVYA, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.998843783 (81h49m46.869926818s) [noise=true , address=77.90.31.18:4830 , piece_id=3CTHRQHH7RNBOS6DDU3PUHE5TGEQMB4UBDSJEXE67GM7PXJIEO4A, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.002758567 (81h49m46.873841602s) [noise=true , address=77.90.31.17:4769 , piece_id=66IBD3RPUO7QFO4ALJSH5W4ELFKPLDOHSMIQ7UWNGGDV3OXZGUQQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.010756346 (81h49m46.881839382s) [noise=true , address=77.90.31.11:4570 , piece_id=HIPMDO4ST4XFQDT3M7PFIIARDMPBX2WEIPRVHOQI4OEIPMJIZZRA, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.088053593 (81h49m46.959136638s) [noise=true , address=45.91.116.10:4098 , piece_id=Q6MKNY37MMBVHUOMWJCTPE5BPXWG7G7RLEHQKLXYSDSI2KKYTPNQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.094063277 (81h49m46.965146322s) [noise=true , address=77.90.31.10:4530 , piece_id=XVCYKZOJ2H3U5RWMWW3BIBW37YK3ZQUHNIJHSGGTP7R2X244Y55Q, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.101953681 (81h49m46.973036726s) [noise=true , address=45.91.116.5:4306 , piece_id=5S4PWZULVPGCP4I4PSINBYKIOCBPLFIUOZE5QO4CKTQ3I7P76BQQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.109207003 (81h49m46.980290038s) [noise=true , address=77.90.31.10:4538 , piece_id=KTOSBWL7QVBHLWCAMOZNTHRCLWMUUHCLIPI2W52KOHNJ55NLNRWQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.109225769 (81h49m46.980308804s) [seg_pos={0 0}, upload=71502]: Segment reached optimal threshold of 14 pieces. Dec 9 01:18:47.109408547 (81h49m46.980491592s) [noise=true , address=77.90.31.15:4713 , piece_id=BCW7S3OAKKGDYPH7RP7B6VJJSG5HNCR2DTLMHZDRLO4EEOPH7FKQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Liv9RcEgfjxodPuadijzkcUGgJEy7LfKqHGFhaJCKbcxKkNN8): protocol: expected piece hash; context canceled; EOF Dec 9 01:18:47.109416322 (81h49m46.980499357s) [noise=true , address=77.90.31.15:4713 , piece_id=BCW7S3OAKKGDYPH7RP7B6VJJSG5HNCR2DTLMHZDRLO4EEOPH7FKQ, seg_pos={0 0}, upload=71502]: Long tail cancellation due to optimal threshold Dec 9 01:18:47.109443945 (81h49m46.98052698s) [noise=true , address=45.91.116.9:4287 , piece_id=GUUNM5MV3J7OELO633ESY3EMGLJOZKXNUKT3SRI7GKYJDYIC6SPA, seg_pos={0 0}, upload=71502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cNuooGmeprkK8fRRBY5ZSAbWda7fdfPbYwbg3fV2TKGsYikwf): protocol: expected piece hash; context canceled; EOF Dec 9 01:18:47.109452361 (81h49m46.980535396s) [noise=true , address=45.91.116.9:4287 , piece_id=GUUNM5MV3J7OELO633ESY3EMGLJOZKXNUKT3SRI7GKYJDYIC6SPA, seg_pos={0 0}, upload=71502]: Long tail cancellation due to optimal threshold Dec 9 01:18:47.109521943 (81h49m46.980604978s) [noise=true , address=145.79.227.4:4205 , piece_id=BWNYAT67IELSPLQTYMAM2O76OLQWUTXACQXR4VZSAQE67NYOASRA, seg_pos={0 0}, upload=71502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pih2cgjPo5YeP6B7g8diM7p11VRJfHy5H34GUT9CK77W2cTLr): protocol: expected piece hash; context canceled; EOF Dec 9 01:18:47.109527924 (81h49m46.980610969s) [noise=true , address=145.79.227.4:4205 , piece_id=BWNYAT67IELSPLQTYMAM2O76OLQWUTXACQXR4VZSAQE67NYOASRA, seg_pos={0 0}, upload=71502]: Long tail cancellation due to optimal threshold Dec 9 01:18:47.109629788 (81h49m46.980712823s) [noise=true , address=77.90.31.16:4736 , piece_id=BHZSQ7MXD2X4YEGL6IYMUYCHURUXDRRRJDYPTBVZ44SORXMPSO5A, seg_pos={0 0}, upload=71502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: expected piece hash; context canceled; EOF Dec 9 01:18:47.109635399 (81h49m46.980718444s) [noise=true , address=77.90.31.16:4736 , piece_id=BHZSQ7MXD2X4YEGL6IYMUYCHURUXDRRRJDYPTBVZ44SORXMPSO5A, seg_pos={0 0}, upload=71502]: Long tail cancellation due to optimal threshold Dec 9 01:18:47.109669504 (81h49m46.980752549s) [seg_pos={0 0}, upload=71502]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 9 01:18:47.109759546 (81h49m46.980842591s) [upload=71502]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 01:18:47.109766839 (81h49m46.980849874s) [upload=71502]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 01:19:58.768357796 (81h50m58.639440831s) [upload=71502]: Done uploading segments... Dec 9 01:19:58.768409825 (81h50m58.63949286s) [upload=71502]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.944376217s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [...X........X............X.X................................] duration: 5.982721776s trace: https://jaeger.dev.storj.io/trace/4a1f040f80b86635 FAIL (2025-12-09 01:51:43 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.982747343s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..............................................X.............] duration: 3.450170132s trace: https://jaeger.dev.storj.io/trace/1a943f69719bbfc6 FAIL (2025-12-09 01:51:13 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.450199188s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [....X........X............X.X...............................] duration: 5.875820189s trace: https://jaeger.dev.storj.io/trace/4a1f040f80b86635 FAIL (2025-12-09 01:50:54 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.875929918s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 2.055075794s trace: https://jaeger.dev.storj.io/trace/36455c540a91ba90 FAIL (2025-12-09 01:50:53 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 2.055108315s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 6.118429487s trace: https://jaeger.dev.storj.io/trace/1b9ac1acdb153d00 FAIL (2025-12-09 01:50:35 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== A, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.018287744 (81h49m44.88937079s) [noise=true , address=45.91.116.10:4098 , piece_id=Q6MKNY37MMBVHUOMWJCTPE5BPXWG7G7RLEHQKLXYSDSI2KKYTPNQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.018979372 (81h49m44.890062467s) [noise=true , address=45.91.116.5:4306 , piece_id=5S4PWZULVPGCP4I4PSINBYKIOCBPLFIUOZE5QO4CKTQ3I7P76BQQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.019582951 (81h49m44.890666006s) [noise=true , address=77.90.31.10:4538 , piece_id=KTOSBWL7QVBHLWCAMOZNTHRCLWMUUHCLIPI2W52KOHNJ55NLNRWQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.019819150 (81h49m44.890902196s) [noise=true , address=77.90.31.10:4530 , piece_id=XVCYKZOJ2H3U5RWMWW3BIBW37YK3ZQUHNIJHSGGTP7R2X244Y55Q, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.020245322 (81h49m44.891328377s) [noise=true , address=45.91.116.9:4287 , piece_id=GUUNM5MV3J7OELO633ESY3EMGLJOZKXNUKT3SRI7GKYJDYIC6SPA, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.020660623 (81h49m44.891743658s) [noise=true , address=145.79.227.12:4769 , piece_id=YTUOFFOTJBPKJUKCRDIY6S65ZP3W5XJCW7GU6GLSFHDUOPOTQO4Q, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.021074250 (81h49m44.892157295s) [noise=true , address=77.90.31.11:4570 , piece_id=HIPMDO4ST4XFQDT3M7PFIIARDMPBX2WEIPRVHOQI4OEIPMJIZZRA, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.021462369 (81h49m44.892545414s) [noise=true , address=145.79.227.49:4359 , piece_id=FDOP6PX5XFB5OJVC7H4S6ZFY4FT5AAWOLIH43M4GTLCZS5YKIM5Q, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.021981778 (81h49m44.893064813s) [noise=true , address=145.79.227.52:4454 , piece_id=7YBPVY5EXGAIQKJEP5TCXX5KRFBUBC45QQI6CONCS5LGUMRU3Y4Q, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.022307669 (81h49m44.893390704s) [noise=true , address=145.79.227.9:4384 , piece_id=3JXASIXRKWJ573Q466LHQLNOZXF4UC2A6LRJKZE2KBOY2N444EUA, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.022633470 (81h49m44.893716505s) [noise=true , address=64.49.13.170:4722 , piece_id=Y5ZZFYYBICQCKHXREFINQ6LA6IUU4MAS4OA4SJXFBC2BFMIG4IMQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.023012681 (81h49m44.894095716s) [noise=true , address=77.90.31.16:4755 , piece_id=M2EBCSYLP54HIYVMV2WFXQRTLSISUJKZPGQRB6K6EEWSRKJKWVYA, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.023282415 (81h49m44.89436546s) [noise=true , address=145.79.227.4:4205 , piece_id=BWNYAT67IELSPLQTYMAM2O76OLQWUTXACQXR4VZSAQE67NYOASRA, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.023555585 (81h49m44.89463862s) [noise=true , address=77.90.31.16:4736 , piece_id=BHZSQ7MXD2X4YEGL6IYMUYCHURUXDRRRJDYPTBVZ44SORXMPSO5A, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.023817675 (81h49m44.89490071s) [noise=true , address=77.90.31.15:4713 , piece_id=BCW7S3OAKKGDYPH7RP7B6VJJSG5HNCR2DTLMHZDRLO4EEOPH7FKQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.024446281 (81h49m44.895529327s) [noise=true , address=77.90.31.17:4769 , piece_id=66IBD3RPUO7QFO4ALJSH5W4ELFKPLDOHSMIQ7UWNGGDV3OXZGUQQ, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:45.024703942 (81h49m44.895786987s) [noise=true , address=145.79.227.39:4037 , piece_id=PGBSSKDC7LGLKZE3FGX6FHAC6H45RMOLJFVQHCJPWWAVZSOSIG5A, seg_pos={0 0}, upload=71502]: Uploading piece... Dec 9 01:18:46.437570978 (81h49m46.308654013s) [upload=71502]: Next returned nil segment Dec 9 01:18:46.437586438 (81h49m46.308669483s) [upload=71502]: Waiting for error group managing segments... Dec 9 01:18:46.796243098 (81h49m46.667326134s) [noise=true , address=64.49.13.170:4722 , piece_id=Y5ZZFYYBICQCKHXREFINQ6LA6IUU4MAS4OA4SJXFBC2BFMIG4IMQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.894539060 (81h49m46.765622105s) [noise=true , address=145.79.227.39:4037 , piece_id=PGBSSKDC7LGLKZE3FGX6FHAC6H45RMOLJFVQHCJPWWAVZSOSIG5A, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.906718262 (81h49m46.777801297s) [noise=true , address=145.79.227.9:4384 , piece_id=3JXASIXRKWJ573Q466LHQLNOZXF4UC2A6LRJKZE2KBOY2N444EUA, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.914490982 (81h49m46.785574027s) [noise=true , address=145.79.227.49:4359 , piece_id=FDOP6PX5XFB5OJVC7H4S6ZFY4FT5AAWOLIH43M4GTLCZS5YKIM5Q, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.928206178 (81h49m46.799289213s) [noise=true , address=145.79.227.52:4454 , piece_id=7YBPVY5EXGAIQKJEP5TCXX5KRFBUBC45QQI6CONCS5LGUMRU3Y4Q, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.953023966 (81h49m46.824107001s) [noise=true , address=145.79.227.12:4769 , piece_id=YTUOFFOTJBPKJUKCRDIY6S65ZP3W5XJCW7GU6GLSFHDUOPOTQO4Q, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.994863524 (81h49m46.865946569s) [noise=true , address=77.90.31.16:4755 , piece_id=M2EBCSYLP54HIYVMV2WFXQRTLSISUJKZPGQRB6K6EEWSRKJKWVYA, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:46.998843783 (81h49m46.869926818s) [noise=true , address=77.90.31.18:4830 , piece_id=3CTHRQHH7RNBOS6DDU3PUHE5TGEQMB4UBDSJEXE67GM7PXJIEO4A, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.002758567 (81h49m46.873841602s) [noise=true , address=77.90.31.17:4769 , piece_id=66IBD3RPUO7QFO4ALJSH5W4ELFKPLDOHSMIQ7UWNGGDV3OXZGUQQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.010756346 (81h49m46.881839382s) [noise=true , address=77.90.31.11:4570 , piece_id=HIPMDO4ST4XFQDT3M7PFIIARDMPBX2WEIPRVHOQI4OEIPMJIZZRA, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.088053593 (81h49m46.959136638s) [noise=true , address=45.91.116.10:4098 , piece_id=Q6MKNY37MMBVHUOMWJCTPE5BPXWG7G7RLEHQKLXYSDSI2KKYTPNQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.094063277 (81h49m46.965146322s) [noise=true , address=77.90.31.10:4530 , piece_id=XVCYKZOJ2H3U5RWMWW3BIBW37YK3ZQUHNIJHSGGTP7R2X244Y55Q, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.101953681 (81h49m46.973036726s) [noise=true , address=45.91.116.5:4306 , piece_id=5S4PWZULVPGCP4I4PSINBYKIOCBPLFIUOZE5QO4CKTQ3I7P76BQQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.109207003 (81h49m46.980290038s) [noise=true , address=77.90.31.10:4538 , piece_id=KTOSBWL7QVBHLWCAMOZNTHRCLWMUUHCLIPI2W52KOHNJ55NLNRWQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: <nil> Dec 9 01:18:47.109225769 (81h49m46.980308804s) [seg_pos={0 0}, upload=71502]: Segment reached optimal threshold of 14 pieces. Dec 9 01:18:47.109408547 (81h49m46.980491592s) [noise=true , address=77.90.31.15:4713 , piece_id=BCW7S3OAKKGDYPH7RP7B6VJJSG5HNCR2DTLMHZDRLO4EEOPH7FKQ, seg_pos={0 0}, upload=71502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Liv9RcEgfjxodPuadijzkcUGgJEy7LfKqHGFhaJCKbcxKkNN8): protocol: expected piece hash; context canceled; EOF Dec 9 01:18:47.109416322 (81h49m46.980499357s) [noise=true , address=77.90.31.15:4713 , piece_id=BCW7S3OAKKGDYPH7RP7B6VJJSG5HNCR2DTLMHZDRLO4EEOPH7FKQ, seg_pos={0 0}, upload=71502]: Long tail cancellation due to optimal threshold Dec 9 01:18:47.109443945 (81h49m46.98052698s) [noise=true , address=45.91.116.9:4287 , piece_id=GUUNM5MV3J7OELO633ESY3EMGLJOZKXNUKT3SRI7GKYJDYIC6SPA, seg_pos={0 0}, upload=71502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cNuooGmeprkK8fRRBY5ZSAbWda7fdfPbYwbg3fV2TKGsYikwf): protocol: expected piece hash; context canceled; EOF Dec 9 01:18:47.109452361 (81h49m46.980535396s) [noise=true , address=45.91.116.9:4287 , piece_id=GUUNM5MV3J7OELO633ESY3EMGLJOZKXNUKT3SRI7GKYJDYIC6SPA, seg_pos={0 0}, upload=71502]: Long tail cancellation due to optimal threshold Dec 9 01:18:47.109521943 (81h49m46.980604978s) [noise=true , address=145.79.227.4:4205 , piece_id=BWNYAT67IELSPLQTYMAM2O76OLQWUTXACQXR4VZSAQE67NYOASRA, seg_pos={0 0}, upload=71502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pih2cgjPo5YeP6B7g8diM7p11VRJfHy5H34GUT9CK77W2cTLr): protocol: expected piece hash; context canceled; EOF Dec 9 01:18:47.109527924 (81h49m46.980610969s) [noise=true , address=145.79.227.4:4205 , piece_id=BWNYAT67IELSPLQTYMAM2O76OLQWUTXACQXR4VZSAQE67NYOASRA, seg_pos={0 0}, upload=71502]: Long tail cancellation due to optimal threshold Dec 9 01:18:47.109629788 (81h49m46.980712823s) [noise=true , address=77.90.31.16:4736 , piece_id=BHZSQ7MXD2X4YEGL6IYMUYCHURUXDRRRJDYPTBVZ44SORXMPSO5A, seg_pos={0 0}, upload=71502]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: expected piece hash; context canceled; EOF Dec 9 01:18:47.109635399 (81h49m46.980718444s) [noise=true , address=77.90.31.16:4736 , piece_id=BHZSQ7MXD2X4YEGL6IYMUYCHURUXDRRRJDYPTBVZ44SORXMPSO5A, seg_pos={0 0}, upload=71502]: Long tail cancellation due to optimal threshold Dec 9 01:18:47.109669504 (81h49m46.980752549s) [seg_pos={0 0}, upload=71502]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 9 01:18:47.109759546 (81h49m46.980842591s) [upload=71502]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 01:18:47.109766839 (81h49m46.980849874s) [upload=71502]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 01:19:58.768357796 (81h50m58.639440831s) [upload=71502]: Done uploading segments... Dec 9 01:19:58.768409825 (81h50m58.63949286s) [upload=71502]: Upload finished. err: metaclient: context deadline exceeded Duration: 6.118458781s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...............................................X............] duration: 3.356328532s trace: https://jaeger.dev.storj.io/trace/1a943f69719bbfc6 FAIL (2025-12-09 01:50:33 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.356362045s core uplink us-select-1 us1-placement-20

1
...
11648
11649
11650
11651
11652
...
17335