Orbiter History

checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 5.761393934s trace: https://jaeger.dev.storj.io/trace/5b0e6327850e4d93 FAIL (2025-12-10 23:41:43 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.761424542s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..........................X.................................] duration: 4.070134142s trace: https://jaeger.dev.storj.io/trace/30cdf86753ce10d1 FAIL (2025-12-10 23:41:27 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 4.070157055s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 5.763489341s trace: https://jaeger.dev.storj.io/trace/5b0e6327850e4d93 FAIL (2025-12-10 23:41:24 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.763517546s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 16 out of 60 is failed (threshold=3) [........................................XXXXX.XXXXX.X..XXXXX] duration: 1m0.248799022s trace: https://jaeger.dev.storj.io/trace/74dfd5427c294c1 FAIL (2025-12-10 23:41:07 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 1940]: Done starting upload Dec 10 23:40:07.656039304 (128h11m7.527122349s) [upload=111940]: Uploading segments... Dec 10 23:40:07.656149615 (128h11m7.52723266s) [upload=111940]: Got next segment. Inline: false Dec 10 23:40:07.656197456 (128h11m7.527280511s) [upload=111940]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 10 23:40:07.656204850 (128h11m7.527287895s) [upload=111940]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 10 23:40:07.903660540 (128h11m7.774743595s) [seg_pos={0 0}, upload=111940]: Begin upload segment... Dec 10 23:40:07.903745591 (128h11m7.774828626s) [seg_pos={0 0}, upload=111940]: Done begin upload segment. Dec 10 23:40:07.903803742 (128h11m7.774886787s) [noise=true , address=5.231.73.5:4026 , piece_id=LH6LU2J5W55O6BOKSULNSSGIM52WILRHD7SM4QSV74ZRKRKUJYKQ, seg_pos={0 0}, upload=111940]: Uploading piece... Dec 10 23:40:07.904615467 (128h11m7.775698523s) [noise=true , address=77.90.31.19:4857 , piece_id=A235IXSDN55MOPQMLSJDI5Q4RQSSKRCDJAFCY5WBD4LSRDQNMYYQ, seg_pos={0 0}, upload=111940]: Uploading piece... Dec 10 23:40:07.905558954 (128h11m7.776641999s) [noise=true , address=77.90.31.11:4559 , piece_id=HKJQJS3K5XRG6JB5EDZNEUQY5TRRRPDUP56KC4GF7Y6RLOS4FSDA, seg_pos={0 0}, upload=111940]: Uploading piece... Dec 10 23:40:07.906402209 (128h11m7.777485254s) [noise=true , address=145.79.227.44:4217 , piece_id=UVOHYA54GAVKHZOINSYIDX7OEP5U7EORNSMDQZVPAOZWIGJX244Q, seg_pos={0 0}, upload=111940]: Uploading piece... Dec 10 23:40:07.907095610 (128h11m7.778178655s) [noise=true , address=145.79.227.3:4177 , piece_id=NJYJP3AMHRTI36EDDFFP5NU5KVSXWHPVEQZIMFYLFPU4I55FPZKQ, seg_pos={0 0}, upload=111940]: Uploading piece... Dec 10 23:40:07.907884522 (128h11m7.778967567s) [noise=true , address=77.90.31.17:4796 , piece_id=XH63AUK2DYN3NDAOG7HNXMSB4AGX6QGCS2QTD2ZVNV42QLFQ2OBA, seg_pos={0 0}, upload=111940]: Uploading piece... Dec 10 23:40:07.908589024 (128h11m7.779672059s) [noise=true , address=45.91.116.7:4018 , piece_id=FM7X2HU2MHJDARXA65XKCMXOGORR62ZUL6HUFVN4RM6Y5AUBREDA, seg_pos={0 0}, upload=111940]: Uploading piece... Dec 10 23:40:07.909361114 (128h11m7.780444159s) [noise=true , address=45.91.116.4:4948 , piece_id=G5AUHEDAVGYDDNKV5VAH6XONKS2AHVK2LI3JYO63CEW7RC5YLRZA, seg_pos={0 0}, upload=111940]: Uploading piece... Dec 10 23:40:07.910166026 (128h11m7.781249071s) [noise=true , address=193.149.164.143:4699 , piece_id=ZY2NNT5DELDNTIJNWN27OXW3Z2FLYR7VQ2T7NRMMGPTPNLPROCNA, seg_pos={0 0}, upload=111940]: Uploading piece... Dec 10 23:40:07.912503768 (128h11m7.783586813s) [noise=true , address=145.79.227.14:4853 , piece_id=4OY4YDB4FCOCX2SXZPMGYFWRFUGZMBKUUSS4UFVKUTIZWKLNGGHQ, seg_pos={0 0}, upload=111940]: Uploading piece... Dec 10 23:40:07.914815049 (128h11m7.785898094s) [noise=true , address=45.91.116.4:4968 , piece_id=PHV6ZIGZMPMKZWOC6QPZOEZ2Z3G2EBMTOGE4XQVMAFGXO4CXHPBA, seg_pos={0 0}, upload=111940]: Uploading piece... Dec 10 23:40:07.916962868 (128h11m7.788045913s) [noise=true , address=45.91.116.11:4134 , piece_id=B3WYAYIGVCQUL5DIYK2LY65ITYKGV44MK42R3H36K57NPKG76F4A, seg_pos={0 0}, upload=111940]: Uploading piece... Dec 10 23:40:07.918888646 (128h11m7.789971691s) [noise=true , address=145.79.227.50:4399 , piece_id=DYYE46C6MEWEX3IYNGHIPB7ZN3A5NF35B4YGIJWRV3EDWGU26Y3Q, seg_pos={0 0}, upload=111940]: Uploading piece... Dec 10 23:40:07.920786930 (128h11m7.791869975s) [noise=true , address=45.91.116.14:4201 , piece_id=OKLF4VYSOL7CMPCZPHYSEUDNSMV46HVKBERL3WUGN7HNYL327FTA, seg_pos={0 0}, upload=111940]: Uploading piece... Dec 10 23:40:07.922938097 (128h11m7.794021142s) [noise=true , address=77.90.31.16:4743 , piece_id=36PFPHGS22QDZEXVOAQWOEXAB5DSOD6JCFZK6EHYWNJEHCAIZGXQ, seg_pos={0 0}, upload=111940]: Uploading piece... Dec 10 23:40:07.923903855 (128h11m7.7949869s) [noise=true , address=45.91.116.9:4267 , piece_id=E7PWH6ZVWOKRL7GBQ372PREHLIROYF2EH7SXLOFLPZBYNPXI2TEQ, seg_pos={0 0}, upload=111940]: Uploading piece... Dec 10 23:40:07.925956002 (128h11m7.797039048s) [noise=true , address=77.90.31.15:4714 , piece_id=JOKGVBPSEQA2CTCR4W2S7YNNCJ2AVGXIDNIBHB3XE2WBPBZZTS7A, seg_pos={0 0}, upload=111940]: Uploading piece... Dec 10 23:40:07.928913865 (128h11m7.79999695s) [noise=true , address=145.79.227.30:4793 , piece_id=GGSGMYS7V7CTVECT3VXUZUBA6MEYBFWUS5PR2TTTPWHAZ5VSVWUA, seg_pos={0 0}, upload=111940]: Uploading piece... Dec 10 23:40:09.226202530 (128h11m9.097285575s) [upload=111940]: Next returned nil segment Dec 10 23:40:09.226217177 (128h11m9.097300222s) [upload=111940]: Waiting for error group managing segments... Dec 10 23:40:09.558148088 (128h11m9.429231124s) [noise=true , address=193.149.164.143:4699 , piece_id=ZY2NNT5DELDNTIJNWN27OXW3Z2FLYR7VQ2T7NRMMGPTPNLPROCNA, seg_pos={0 0}, upload=111940]: Done uploading piece. err: <nil> Dec 10 23:40:09.774681498 (128h11m9.645764533s) [noise=true , address=145.79.227.50:4399 , piece_id=DYYE46C6MEWEX3IYNGHIPB7ZN3A5NF35B4YGIJWRV3EDWGU26Y3Q, seg_pos={0 0}, upload=111940]: Done uploading piece. err: <nil> Dec 10 23:40:09.788244764 (128h11m9.659327799s) [noise=true , address=145.79.227.3:4177 , piece_id=NJYJP3AMHRTI36EDDFFP5NU5KVSXWHPVEQZIMFYLFPU4I55FPZKQ, seg_pos={0 0}, upload=111940]: Done uploading piece. err: <nil> Dec 10 23:40:09.813655009 (128h11m9.684738044s) [noise=true , address=145.79.227.14:4853 , piece_id=4OY4YDB4FCOCX2SXZPMGYFWRFUGZMBKUUSS4UFVKUTIZWKLNGGHQ, seg_pos={0 0}, upload=111940]: Done uploading piece. err: <nil> Dec 10 23:40:09.840430513 (128h11m9.711513548s) [noise=true , address=5.231.73.5:4026 , piece_id=LH6LU2J5W55O6BOKSULNSSGIM52WILRHD7SM4QSV74ZRKRKUJYKQ, seg_pos={0 0}, upload=111940]: Done uploading piece. err: <nil> Dec 10 23:40:09.911674661 (128h11m9.782757697s) [noise=true , address=45.91.116.4:4968 , piece_id=PHV6ZIGZMPMKZWOC6QPZOEZ2Z3G2EBMTOGE4XQVMAFGXO4CXHPBA, seg_pos={0 0}, upload=111940]: Done uploading piece. err: <nil> Dec 10 23:40:09.922324149 (128h11m9.793407204s) [noise=true , address=45.91.116.4:4948 , piece_id=G5AUHEDAVGYDDNKV5VAH6XONKS2AHVK2LI3JYO63CEW7RC5YLRZA, seg_pos={0 0}, upload=111940]: Done uploading piece. err: <nil> Dec 10 23:40:09.948082627 (128h11m9.819165672s) [noise=true , address=45.91.116.7:4018 , piece_id=FM7X2HU2MHJDARXA65XKCMXOGORR62ZUL6HUFVN4RM6Y5AUBREDA, seg_pos={0 0}, upload=111940]: Done uploading piece. err: <nil> Dec 10 23:40:09.953476047 (128h11m9.824559092s) [noise=true , address=45.91.116.9:4267 , piece_id=E7PWH6ZVWOKRL7GBQ372PREHLIROYF2EH7SXLOFLPZBYNPXI2TEQ, seg_pos={0 0}, upload=111940]: Done uploading piece. err: <nil> Dec 10 23:40:09.969572346 (128h11m9.840655392s) [noise=true , address=45.91.116.11:4134 , piece_id=B3WYAYIGVCQUL5DIYK2LY65ITYKGV44MK42R3H36K57NPKG76F4A, seg_pos={0 0}, upload=111940]: Done uploading piece. err: <nil> Dec 10 23:40:09.990630965 (128h11m9.86171401s) [noise=true , address=45.91.116.14:4201 , piece_id=OKLF4VYSOL7CMPCZPHYSEUDNSMV46HVKBERL3WUGN7HNYL327FTA, seg_pos={0 0}, upload=111940]: Done uploading piece. err: <nil> Dec 10 23:40:10.770644196 (128h11m10.641727241s) [noise=true , address=145.79.227.44:4217 , piece_id=UVOHYA54GAVKHZOINSYIDX7OEP5U7EORNSMDQZVPAOZWIGJX244Q, seg_pos={0 0}, upload=111940]: Done uploading piece. err: <nil> Dec 10 23:40:10.784742371 (128h11m10.655825416s) [noise=true , address=145.79.227.30:4793 , piece_id=GGSGMYS7V7CTVECT3VXUZUBA6MEYBFWUS5PR2TTTPWHAZ5VSVWUA, seg_pos={0 0}, upload=111940]: Done uploading piece. err: <nil> Dec 10 23:41:06.969952547 (128h12m6.841035582s) [noise=true , address=77.90.31.19:4857 , piece_id=A235IXSDN55MOPQMLSJDI5Q4RQSSKRCDJAFCY5WBD4LSRDQNMYYQ, seg_pos={0 0}, upload=111940]: Done uploading piece. err: ecclient: upload failed (node:12i57FF97oEddYLZpNxSq5or9WLEquL8ZW1cVUuW4AThBbDqcCL, address:77.90.31.19:4857): protocol: context deadline exceeded; context deadline exceeded Dec 10 23:41:06.970152357 (128h12m6.841235392s) [noise=true , address=77.90.31.15:4714 , piece_id=JOKGVBPSEQA2CTCR4W2S7YNNCJ2AVGXIDNIBHB3XE2WBPBZZTS7A, seg_pos={0 0}, upload=111940]: Done uploading piece. err: ecclient: upload failed (node:12DTMJVUhaPKvEnvujCAk9byGb88QsevLbtCBB9R1mxS4DxShiP, address:77.90.31.15:4714): protocol: rpc: dial tcp 77.90.31.15:4714: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:41:06.970343701 (128h12m6.841426747s) [noise=true , address=77.90.31.11:4559 , piece_id=HKJQJS3K5XRG6JB5EDZNEUQY5TRRRPDUP56KC4GF7Y6RLOS4FSDA, seg_pos={0 0}, upload=111940]: Done uploading piece. err: ecclient: upload failed (node:1KER2FucoaWotduZNiFAnbFz7gotJYFofKnyQNCRvWQ8rjVDwY, address:77.90.31.11:4559): protocol: write tcp 188.166.225.170:50610->77.90.31.11:4559: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:41:06.970364851 (128h12m6.841447896s) [noise=true , address=77.90.31.17:4796 , piece_id=XH63AUK2DYN3NDAOG7HNXMSB4AGX6QGCS2QTD2ZVNV42QLFQ2OBA, seg_pos={0 0}, upload=111940]: Done uploading piece. err: ecclient: upload failed (node:1KK6y5NBuZovne8n3gMvAyDcZFasZxDzorvPeHe7CefoFs27UF, address:77.90.31.17:4796): protocol: rpc: dial tcp 77.90.31.17:4796: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:41:06.970430246 (128h12m6.841513291s) [noise=true , address=77.90.31.16:4743 , piece_id=36PFPHGS22QDZEXVOAQWOEXAB5DSOD6JCFZK6EHYWNJEHCAIZGXQ, seg_pos={0 0}, upload=111940]: Done uploading piece. err: ecclient: upload failed (node:12XLPBKV4R1CLgVK32nSQffKHxjhU7XsGo8MEsHaAbDGiusTTAv, address:77.90.31.16:4743): protocol: rpc: dial tcp 77.90.31.16:4743: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:41:06.970456125 (128h12m6.84153917s) [seg_pos={0 0}, upload=111940]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Dec 10 23:41:07.216937980 (128h12m7.088021015s) [upload=111940]: Done uploading segments... Dec 10 23:41:07.216982905 (128h12m7.08806595s) [upload=111940]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1m0.248832735s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...........................X................................] duration: 4.140130684s trace: https://jaeger.dev.storj.io/trace/30cdf86753ce10d1 FAIL (2025-12-10 23:41:07 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 4.140184176s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [............................X...............................] duration: 4.052712368s trace: https://jaeger.dev.storj.io/trace/30cdf86753ce10d1 FAIL (2025-12-10 23:40:51 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 4.052740011s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 6.055426699s trace: https://jaeger.dev.storj.io/trace/5b0e6327850e4d93 FAIL (2025-12-10 23:40:22 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.05545386s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 15 out of 60 is failed (threshold=3) [.........................................XXXXX.XXXXX.X..XXXX] duration: 1m0.244104077s trace: https://jaeger.dev.storj.io/trace/724700451675715 FAIL (2025-12-10 23:40:06 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== ush batch item: pb.BatchRequestItem_ObjectBegin Dec 10 23:39:07.182225901 (128h10m7.053308936s) [upload=111928]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 10 23:39:07.433584401 (128h10m7.304667436s) [seg_pos={0 0}, upload=111928]: Begin upload segment... Dec 10 23:39:07.433738444 (128h10m7.304821479s) [noise=true , address=145.79.227.48:4320 , piece_id=SEV7POKAHXFANDZ665PHZSD6YDR3V7FMDPXB5ZMVGDU23NDY6FUA, seg_pos={0 0}, upload=111928]: Uploading piece... Dec 10 23:39:07.434359356 (128h10m7.305442401s) [seg_pos={0 0}, upload=111928]: Done begin upload segment. Dec 10 23:39:07.434393672 (128h10m7.305476717s) [noise=true , address=77.90.31.18:4817 , piece_id=IN4RN2HSPAFOBKF222CABNACKMEEIAISZGXUIBIRIQYNXKSW25HQ, seg_pos={0 0}, upload=111928]: Uploading piece... Dec 10 23:39:07.435244482 (128h10m7.306327527s) [noise=true , address=77.90.31.12:4603 , piece_id=UIUTDBXBEFZLLYKFZOHPCKD445KGLD7OD24EEOIQEKC2FZV6QQ2A, seg_pos={0 0}, upload=111928]: Uploading piece... Dec 10 23:39:07.435627301 (128h10m7.306710346s) [noise=true , address=77.90.31.15:4714 , piece_id=TTJCXS4GIWNLT6X5FVCGVDCXYVYA7QLXZW3BHVI4Y4RV3T524BQQ, seg_pos={0 0}, upload=111928]: Uploading piece... Dec 10 23:39:07.435990703 (128h10m7.307073738s) [noise=true , address=64.49.13.140:4508 , piece_id=2QHNSAZRXMMJ355LRE7RAC5625TLAH5QJA3WBNPD72V46MCJNUCA, seg_pos={0 0}, upload=111928]: Uploading piece... Dec 10 23:39:07.436387128 (128h10m7.307470163s) [noise=true , address=77.90.31.21:4905 , piece_id=XMRNECERZQACDKO3KUYPFJZNZD56M67H667H4Y6ZNEUASETJZTMQ, seg_pos={0 0}, upload=111928]: Uploading piece... Dec 10 23:39:07.436772101 (128h10m7.307855146s) [noise=true , address=145.79.227.50:4405 , piece_id=DOO4NJECQFLVGL7BXPMFZI2MQAINSG7HLSJFYKQM74ISPPRHQTBQ, seg_pos={0 0}, upload=111928]: Uploading piece... Dec 10 23:39:07.437128289 (128h10m7.308211324s) [noise=true , address=145.79.227.45:4257 , piece_id=S7AEPGXA3MDRWTVNQGDXCWZYPZBXKG5G4XNK33NAIWKS6QPPMRJQ, seg_pos={0 0}, upload=111928]: Uploading piece... Dec 10 23:39:07.437494226 (128h10m7.308577271s) [noise=true , address=77.90.31.13:4635 , piece_id=IIVTKPI5SCTZWODAE2HBI5GABYYETIL4VKOM2W2GAQAOMII5WG5A, seg_pos={0 0}, upload=111928]: Uploading piece... Dec 10 23:39:07.437931940 (128h10m7.309014985s) [noise=true , address=45.91.116.8:4058 , piece_id=AQLDF5WQSUENUYTVFQNZUUCXFJOXI324U6GY2LN4ARM4QAFN4EAQ, seg_pos={0 0}, upload=111928]: Uploading piece... Dec 10 23:39:07.437984870 (128h10m7.309067915s) [noise=true , address=5.231.73.3:4029 , piece_id=ZOK4Z2QMDTOQAQHB5LLNX2T3TELGGE7OBW6RVWER7NTCO65ZGQGA, seg_pos={0 0}, upload=111928]: Uploading piece... Dec 10 23:39:07.438645668 (128h10m7.309728703s) [noise=true , address=45.91.116.9:4271 , piece_id=YU7LQJFGREN22DFZQ2MKG7JIFR3SBOY54SIHFOGKP7KZNE6QHYEA, seg_pos={0 0}, upload=111928]: Uploading piece... Dec 10 23:39:07.439365028 (128h10m7.310448073s) [noise=true , address=145.79.227.6:4280 , piece_id=YMPINPHBE5OKWQZZNB7RWTXTKLVT2RZ36U7WV2AFXBKNKUE44VPQ, seg_pos={0 0}, upload=111928]: Uploading piece... Dec 10 23:39:07.439828070 (128h10m7.310911105s) [noise=true , address=5.231.73.5:4022 , piece_id=EDAG6KVRHTZAZFBC4G5I5LWSJFMFKPFEIHHE4FHNASQPARDR36YA, seg_pos={0 0}, upload=111928]: Uploading piece... Dec 10 23:39:07.440008494 (128h10m7.311091529s) [noise=true , address=45.91.116.10:4107 , piece_id=QJLU2GYKPJMWP6CGQUIVXHM5HTRHKOKUVSUG4QYUPJVALFRTCCTQ, seg_pos={0 0}, upload=111928]: Uploading piece... Dec 10 23:39:07.440408014 (128h10m7.311491059s) [noise=true , address=77.90.31.16:4758 , piece_id=2GM5764LHBD2K2YVSNVBB7QUJBF3HMPDUD327LVP7YAVCLPMK6VA, seg_pos={0 0}, upload=111928]: Uploading piece... Dec 10 23:39:07.440796053 (128h10m7.311879088s) [noise=true , address=45.91.116.7:4033 , piece_id=GLKPKYPVFRLZ6EVC5ME2NUS3HNFUKT5IWWTQXX3JZNSJJERZLQNQ, seg_pos={0 0}, upload=111928]: Uploading piece... Dec 10 23:39:07.441120371 (128h10m7.312203406s) [noise=true , address=5.231.73.2:4004 , piece_id=6CNKVDADAHDOBHMEY3P6HXDMAWBHNT3TYKGDJKUQSEV7E3Z4IYOA, seg_pos={0 0}, upload=111928]: Uploading piece... Dec 10 23:39:08.723421847 (128h10m8.594504882s) [upload=111928]: Next returned nil segment Dec 10 23:39:08.723436095 (128h10m8.59451914s) [upload=111928]: Waiting for error group managing segments... Dec 10 23:39:09.069781964 (128h10m8.940864999s) [noise=true , address=64.49.13.140:4508 , piece_id=2QHNSAZRXMMJ355LRE7RAC5625TLAH5QJA3WBNPD72V46MCJNUCA, seg_pos={0 0}, upload=111928]: Done uploading piece. err: <nil> Dec 10 23:39:09.292159001 (128h10m9.163242046s) [noise=true , address=145.79.227.48:4320 , piece_id=SEV7POKAHXFANDZ665PHZSD6YDR3V7FMDPXB5ZMVGDU23NDY6FUA, seg_pos={0 0}, upload=111928]: Done uploading piece. err: <nil> Dec 10 23:39:09.307044315 (128h10m9.17812735s) [noise=true , address=145.79.227.50:4405 , piece_id=DOO4NJECQFLVGL7BXPMFZI2MQAINSG7HLSJFYKQM74ISPPRHQTBQ, seg_pos={0 0}, upload=111928]: Done uploading piece. err: <nil> Dec 10 23:39:09.317759477 (128h10m9.188842513s) [noise=true , address=145.79.227.45:4257 , piece_id=S7AEPGXA3MDRWTVNQGDXCWZYPZBXKG5G4XNK33NAIWKS6QPPMRJQ, seg_pos={0 0}, upload=111928]: Done uploading piece. err: <nil> Dec 10 23:39:09.319196123 (128h10m9.190279168s) [noise=true , address=5.231.73.5:4022 , piece_id=EDAG6KVRHTZAZFBC4G5I5LWSJFMFKPFEIHHE4FHNASQPARDR36YA, seg_pos={0 0}, upload=111928]: Done uploading piece. err: <nil> Dec 10 23:39:09.359363928 (128h10m9.230446963s) [noise=true , address=5.231.73.3:4029 , piece_id=ZOK4Z2QMDTOQAQHB5LLNX2T3TELGGE7OBW6RVWER7NTCO65ZGQGA, seg_pos={0 0}, upload=111928]: Done uploading piece. err: <nil> Dec 10 23:39:09.365446720 (128h10m9.236529765s) [noise=true , address=145.79.227.6:4280 , piece_id=YMPINPHBE5OKWQZZNB7RWTXTKLVT2RZ36U7WV2AFXBKNKUE44VPQ, seg_pos={0 0}, upload=111928]: Done uploading piece. err: <nil> Dec 10 23:39:09.377432192 (128h10m9.248515237s) [noise=true , address=5.231.73.2:4004 , piece_id=6CNKVDADAHDOBHMEY3P6HXDMAWBHNT3TYKGDJKUQSEV7E3Z4IYOA, seg_pos={0 0}, upload=111928]: Done uploading piece. err: <nil> Dec 10 23:39:09.412184706 (128h10m9.283267741s) [noise=true , address=45.91.116.10:4107 , piece_id=QJLU2GYKPJMWP6CGQUIVXHM5HTRHKOKUVSUG4QYUPJVALFRTCCTQ, seg_pos={0 0}, upload=111928]: Done uploading piece. err: <nil> Dec 10 23:39:09.428908480 (128h10m9.299991525s) [noise=true , address=45.91.116.7:4033 , piece_id=GLKPKYPVFRLZ6EVC5ME2NUS3HNFUKT5IWWTQXX3JZNSJJERZLQNQ, seg_pos={0 0}, upload=111928]: Done uploading piece. err: <nil> Dec 10 23:39:09.478501963 (128h10m9.349585008s) [noise=true , address=45.91.116.8:4058 , piece_id=AQLDF5WQSUENUYTVFQNZUUCXFJOXI324U6GY2LN4ARM4QAFN4EAQ, seg_pos={0 0}, upload=111928]: Done uploading piece. err: <nil> Dec 10 23:39:09.519150594 (128h10m9.390233629s) [noise=true , address=45.91.116.9:4271 , piece_id=YU7LQJFGREN22DFZQ2MKG7JIFR3SBOY54SIHFOGKP7KZNE6QHYEA, seg_pos={0 0}, upload=111928]: Done uploading piece. err: <nil> Dec 10 23:40:06.495128742 (128h11m6.366211787s) [noise=true , address=77.90.31.18:4817 , piece_id=IN4RN2HSPAFOBKF222CABNACKMEEIAISZGXUIBIRIQYNXKSW25HQ, seg_pos={0 0}, upload=111928]: Done uploading piece. err: ecclient: upload failed (node:12Qv5LBuANP9vS2vbsVNrJ7Mkc3SqXwTNCBzLvN955RHFKQVv5x, address:77.90.31.18:4817): protocol: rpc: dial tcp 77.90.31.18:4817: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:40:06.495371975 (128h11m6.366455011s) [noise=true , address=77.90.31.15:4714 , piece_id=TTJCXS4GIWNLT6X5FVCGVDCXYVYA7QLXZW3BHVI4Y4RV3T524BQQ, seg_pos={0 0}, upload=111928]: Done uploading piece. err: ecclient: upload failed (node:12DTMJVUhaPKvEnvujCAk9byGb88QsevLbtCBB9R1mxS4DxShiP, address:77.90.31.15:4714): protocol: rpc: dial tcp 77.90.31.15:4714: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:40:06.495577066 (128h11m6.366660111s) [noise=true , address=77.90.31.21:4905 , piece_id=XMRNECERZQACDKO3KUYPFJZNZD56M67H667H4Y6ZNEUASETJZTMQ, seg_pos={0 0}, upload=111928]: Done uploading piece. err: ecclient: upload failed (node:1X5A7bLqMEQuaMbf9xZ7PDUD4161sz2QqdM9r59y3bsyPBJhaK, address:77.90.31.21:4905): protocol: rpc: dial tcp 77.90.31.21:4905: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:40:06.495795152 (128h11m6.366878197s) [noise=true , address=77.90.31.13:4635 , piece_id=IIVTKPI5SCTZWODAE2HBI5GABYYETIL4VKOM2W2GAQAOMII5WG5A, seg_pos={0 0}, upload=111928]: Done uploading piece. err: ecclient: upload failed (node:1hkgeCi3efNQFv37chVFEi4v2AyWFEwv9keyy5oz6RkJ4Ubunp, address:77.90.31.13:4635): protocol: rpc: dial tcp 77.90.31.13:4635: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:40:06.496025430 (128h11m6.367108465s) [noise=true , address=77.90.31.16:4758 , piece_id=2GM5764LHBD2K2YVSNVBB7QUJBF3HMPDUD327LVP7YAVCLPMK6VA, seg_pos={0 0}, upload=111928]: Done uploading piece. err: ecclient: upload failed (node:1ahTbggGRi6cJa4JPBfSEgLJYymrgVuLxSc15BV3qj5gPWPFiZ, address:77.90.31.16:4758): protocol: rpc: dial tcp 77.90.31.16:4758: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:40:06.496176909 (128h11m6.367259954s) [noise=true , address=77.90.31.12:4603 , piece_id=UIUTDBXBEFZLLYKFZOHPCKD445KGLD7OD24EEOIQEKC2FZV6QQ2A, seg_pos={0 0}, upload=111928]: Done uploading piece. err: ecclient: upload failed (node:126Sou5pnMm2eE2FJpv6WYX2VSqJvZZf9rFFJD6hFYDetRD28zu, address:77.90.31.12:4603): protocol: write tcp 188.166.225.170:43510->77.90.31.12:4603: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:40:06.496208489 (128h11m6.367291534s) [seg_pos={0 0}, upload=111928]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Dec 10 23:40:06.736121316 (128h11m6.607204361s) [upload=111928]: Done uploading segments... Dec 10 23:40:06.736176852 (128h11m6.607259897s) [upload=111928]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 1m0.244134145s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.............................X..............................] duration: 4.092183476s trace: https://jaeger.dev.storj.io/trace/30cdf86753ce10d1 FAIL (2025-12-10 23:39:49 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 4.092208835s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 5.789557841s trace: https://jaeger.dev.storj.io/trace/5b0e6327850e4d93 FAIL (2025-12-10 23:39:19 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.789585403s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..............................X.............................] duration: 3.856493291s trace: https://jaeger.dev.storj.io/trace/30cdf86753ce10d1 FAIL (2025-12-10 23:39:13 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 3.856525422s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 14 out of 60 is failed (threshold=3) [..........................................XXXXX.XXXXX.X..XXX] duration: 1m0.245937405s trace: https://jaeger.dev.storj.io/trace/72d7b849d012a3d0 FAIL (2025-12-10 23:39:05 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 819 (128h9m5.825809854s) [seg_pos={0 0}, upload=111912]: Done begin upload segment. Dec 10 23:38:05.954754501 (128h9m5.825837546s) [noise=true , address=5.231.73.4:4002 , piece_id=YFFFFZ7EGWYFLGLVY73EWZ33HMLA35W7XOI76JOCHFBDMICLHACA, seg_pos={0 0}, upload=111912]: Uploading piece... Dec 10 23:38:05.954778768 (128h9m5.825861813s) [noise=true , address=77.90.31.21:4885 , piece_id=AX5QUU3ERLBHWWYBBMAFLGIVQWCMQB2NC7NCIZHLBEQO3LDWWPFA, seg_pos={0 0}, upload=111912]: Uploading piece... Dec 10 23:38:05.955306563 (128h9m5.826389608s) [noise=true , address=145.79.227.8:4334 , piece_id=KQLRB574V2JLPMBPP6425PY3WKOMAGUCAHJZEHTZAQAGPIGPKMMA, seg_pos={0 0}, upload=111912]: Uploading piece... Dec 10 23:38:05.955751540 (128h9m5.826834585s) [noise=true , address=5.231.73.3:4009 , piece_id=GXEGVYRWXL5CHLW74WTNP2LLT7VXGVKGB4V5EU4TOS7GM54WVKVQ, seg_pos={0 0}, upload=111912]: Uploading piece... Dec 10 23:38:05.956191708 (128h9m5.827274753s) [noise=true , address=45.91.116.9:4267 , piece_id=MC5SWMXKCI5OJMF7RLULCVWOLPENWYB2CM4DDQUIRJGUWPUHJRDQ, seg_pos={0 0}, upload=111912]: Uploading piece... Dec 10 23:38:05.956750612 (128h9m5.827833647s) [noise=true , address=77.90.31.14:4665 , piece_id=KDVWZGQIVAKK4F3GZB2LBJ2CEB4OFQSNUP5FFNJUTNR56UDYC2EA, seg_pos={0 0}, upload=111912]: Uploading piece... Dec 10 23:38:05.956758127 (128h9m5.827841172s) [noise=true , address=77.90.31.16:4740 , piece_id=4YIRQHRG5NWWJIQI5SHNFVSIJBDREFSW4T3JYU3ECTLPTTU6L6WQ, seg_pos={0 0}, upload=111912]: Uploading piece... Dec 10 23:38:05.957071313 (128h9m5.828154348s) [noise=true , address=45.91.116.8:4055 , piece_id=ZCIBQL5J5ILEFVU277GGY3O26I5SEJVSTUVTKZY5KDG2MS6AB43Q, seg_pos={0 0}, upload=111912]: Uploading piece... Dec 10 23:38:05.957217642 (128h9m5.828300687s) [noise=true , address=45.91.116.5:4301 , piece_id=QIEQABVA64QIT4TSUD4GLCGYRXJ46UVHTLB6ELBMEAHMCZCUH6TQ, seg_pos={0 0}, upload=111912]: Uploading piece... Dec 10 23:38:05.957429936 (128h9m5.828512971s) [noise=true , address=64.49.13.27:4967 , piece_id=KH5CML6SXSUOK5B4G7562Z4LVFJYEKQXKRIG3T62ZFMG4QBR3X2Q, seg_pos={0 0}, upload=111912]: Uploading piece... Dec 10 23:38:05.957668390 (128h9m5.828751435s) [noise=true , address=77.90.31.13:4633 , piece_id=HK47FU577ZNZVAHFGWVMOXADEO26KC2G62ENY4KSE3EVIBY2SMNQ, seg_pos={0 0}, upload=111912]: Uploading piece... Dec 10 23:38:05.957781265 (128h9m5.82886431s) [noise=true , address=45.91.116.6:4994 , piece_id=BUV4W2H4RBQYRPUNWUMM6IVEVVWCZVDY7NPXMUX5YME4PNHHOU6Q, seg_pos={0 0}, upload=111912]: Uploading piece... Dec 10 23:38:05.958210462 (128h9m5.829293507s) [noise=true , address=5.231.73.5:4011 , piece_id=4O6NF5XEGZ72VZWCRSBVJARAOUDCXD4Z4PY3DYRMLDNGWSS3FZEA, seg_pos={0 0}, upload=111912]: Uploading piece... Dec 10 23:38:05.958310542 (128h9m5.829393588s) [noise=true , address=77.90.31.19:4855 , piece_id=ITXGC6WJTWODVZC6SH2BNZJOVULHPZEMYUP5IB4ELCVHG6QKKAYA, seg_pos={0 0}, upload=111912]: Uploading piece... Dec 10 23:38:05.958671310 (128h9m5.829754355s) [noise=true , address=64.49.13.169:4709 , piece_id=ZO3M6OFGFCHNZWJCG2WXOGE26OKJNA5CSGHVYAWVIQJ2SPK4HFMQ, seg_pos={0 0}, upload=111912]: Uploading piece... Dec 10 23:38:05.959009543 (128h9m5.830092588s) [noise=true , address=45.91.116.14:4205 , piece_id=M6S4NIOQR44J2VO4V3TS56CMFJAVOE4O37BKFJJCQIF7NTITK2EQ, seg_pos={0 0}, upload=111912]: Uploading piece... Dec 10 23:38:05.959409355 (128h9m5.8304924s) [noise=true , address=77.90.31.17:4803 , piece_id=NQ42IWZP32XG5UIOYAA2FZVBP2RCOZAP435VQKJJV4ONFNSEQ3LQ, seg_pos={0 0}, upload=111912]: Uploading piece... Dec 10 23:38:05.960553945 (128h9m5.83163698s) [noise=true , address=77.90.31.10:4524 , piece_id=H3JGO5UMD2OJLN6HBLQ6BYGPKYUGZR5ET5RRABMQFVU2JAQ5OP2Q, seg_pos={0 0}, upload=111912]: Uploading piece... Dec 10 23:38:07.251687028 (128h9m7.122770063s) [upload=111912]: Next returned nil segment Dec 10 23:38:07.251702608 (128h9m7.122785653s) [upload=111912]: Waiting for error group managing segments... Dec 10 23:38:07.593345042 (128h9m7.464428077s) [noise=true , address=64.49.13.27:4967 , piece_id=KH5CML6SXSUOK5B4G7562Z4LVFJYEKQXKRIG3T62ZFMG4QBR3X2Q, seg_pos={0 0}, upload=111912]: Done uploading piece. err: <nil> Dec 10 23:38:07.796138969 (128h9m7.667222004s) [noise=true , address=5.231.73.5:4011 , piece_id=4O6NF5XEGZ72VZWCRSBVJARAOUDCXD4Z4PY3DYRMLDNGWSS3FZEA, seg_pos={0 0}, upload=111912]: Done uploading piece. err: <nil> Dec 10 23:38:07.832042484 (128h9m7.70312553s) [noise=true , address=145.79.227.8:4334 , piece_id=KQLRB574V2JLPMBPP6425PY3WKOMAGUCAHJZEHTZAQAGPIGPKMMA, seg_pos={0 0}, upload=111912]: Done uploading piece. err: <nil> Dec 10 23:38:07.839241763 (128h9m7.710324808s) [noise=true , address=5.231.73.3:4009 , piece_id=GXEGVYRWXL5CHLW74WTNP2LLT7VXGVKGB4V5EU4TOS7GM54WVKVQ, seg_pos={0 0}, upload=111912]: Done uploading piece. err: <nil> Dec 10 23:38:07.881714296 (128h9m7.752797351s) [noise=true , address=5.231.73.4:4002 , piece_id=YFFFFZ7EGWYFLGLVY73EWZ33HMLA35W7XOI76JOCHFBDMICLHACA, seg_pos={0 0}, upload=111912]: Done uploading piece. err: <nil> Dec 10 23:38:07.918190272 (128h9m7.789273317s) [noise=true , address=45.91.116.6:4994 , piece_id=BUV4W2H4RBQYRPUNWUMM6IVEVVWCZVDY7NPXMUX5YME4PNHHOU6Q, seg_pos={0 0}, upload=111912]: Done uploading piece. err: <nil> Dec 10 23:38:07.956142979 (128h9m7.827226014s) [noise=true , address=45.91.116.5:4301 , piece_id=QIEQABVA64QIT4TSUD4GLCGYRXJ46UVHTLB6ELBMEAHMCZCUH6TQ, seg_pos={0 0}, upload=111912]: Done uploading piece. err: <nil> Dec 10 23:38:07.964028684 (128h9m7.835111729s) [noise=true , address=45.91.116.8:4055 , piece_id=ZCIBQL5J5ILEFVU277GGY3O26I5SEJVSTUVTKZY5KDG2MS6AB43Q, seg_pos={0 0}, upload=111912]: Done uploading piece. err: <nil> Dec 10 23:38:07.983497826 (128h9m7.854580901s) [noise=true , address=45.91.116.9:4267 , piece_id=MC5SWMXKCI5OJMF7RLULCVWOLPENWYB2CM4DDQUIRJGUWPUHJRDQ, seg_pos={0 0}, upload=111912]: Done uploading piece. err: <nil> Dec 10 23:38:08.007234695 (128h9m7.87831774s) [noise=true , address=45.91.116.14:4205 , piece_id=M6S4NIOQR44J2VO4V3TS56CMFJAVOE4O37BKFJJCQIF7NTITK2EQ, seg_pos={0 0}, upload=111912]: Done uploading piece. err: <nil> Dec 10 23:38:20.925421407 (128h9m20.796504452s) [noise=true , address=64.49.13.169:4709 , piece_id=ZO3M6OFGFCHNZWJCG2WXOGE26OKJNA5CSGHVYAWVIQJ2SPK4HFMQ, seg_pos={0 0}, upload=111912]: Done uploading piece. err: <nil> Dec 10 23:39:05.016141320 (128h10m4.887224365s) [noise=true , address=77.90.31.21:4885 , piece_id=AX5QUU3ERLBHWWYBBMAFLGIVQWCMQB2NC7NCIZHLBEQO3LDWWPFA, seg_pos={0 0}, upload=111912]: Done uploading piece. err: ecclient: upload failed (node:12TfMGrvCMPvQ9ZC1pVoDqjFnUHXtzwwnZB6pCV1em4pdmbK2MP, address:77.90.31.21:4885): protocol: write tcp 188.166.225.170:45838->77.90.31.21:4885: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:39:05.016318047 (128h10m4.887401092s) [noise=true , address=77.90.31.16:4740 , piece_id=4YIRQHRG5NWWJIQI5SHNFVSIJBDREFSW4T3JYU3ECTLPTTU6L6WQ, seg_pos={0 0}, upload=111912]: Done uploading piece. err: ecclient: upload failed (node:145raYQJozNw3SMbj78wDsR6EjPvJTqqcocf1SaPs3HEnNVS5p, address:77.90.31.16:4740): protocol: rpc: dial tcp 77.90.31.16:4740: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:39:05.016623088 (128h10m4.887706133s) [noise=true , address=77.90.31.10:4524 , piece_id=H3JGO5UMD2OJLN6HBLQ6BYGPKYUGZR5ET5RRABMQFVU2JAQ5OP2Q, seg_pos={0 0}, upload=111912]: Done uploading piece. err: ecclient: upload failed (node:12uN7CznoK4AD8FqFkNUndawdGJ9rmjAe7oNnjJkV844ScY87qe, address:77.90.31.10:4524): protocol: write tcp 188.166.225.170:48396->77.90.31.10:4524: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:39:05.016690576 (128h10m4.887773621s) [noise=true , address=77.90.31.17:4803 , piece_id=NQ42IWZP32XG5UIOYAA2FZVBP2RCOZAP435VQKJJV4ONFNSEQ3LQ, seg_pos={0 0}, upload=111912]: Done uploading piece. err: ecclient: upload failed (node:12onacKUY32faZTFuyEjvMCZA4nVKiRKmTKh1WRFMfif2Kw1Q7Y, address:77.90.31.17:4803): protocol: write tcp 188.166.225.170:48496->77.90.31.17:4803: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:39:05.016812279 (128h10m4.887895314s) [noise=true , address=77.90.31.13:4633 , piece_id=HK47FU577ZNZVAHFGWVMOXADEO26KC2G62ENY4KSE3EVIBY2SMNQ, seg_pos={0 0}, upload=111912]: Done uploading piece. err: ecclient: upload failed (node:12qXhyMbfXELW1Vg5z9NpqFX1fm3v2pGEaA56qSc8kHefPNNNYs, address:77.90.31.13:4633): protocol: write tcp 188.166.225.170:45276->77.90.31.13:4633: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:39:05.016853457 (128h10m4.887936492s) [noise=true , address=77.90.31.14:4665 , piece_id=KDVWZGQIVAKK4F3GZB2LBJ2CEB4OFQSNUP5FFNJUTNR56UDYC2EA, seg_pos={0 0}, upload=111912]: Done uploading piece. err: ecclient: upload failed (node:12W4n6uf6azQk1QgeR8Xw7UkpnZm19zdyFKwojCsqZt2GFHrpce, address:77.90.31.14:4665): protocol: context deadline exceeded; context deadline exceeded Dec 10 23:39:05.017021677 (128h10m4.888104712s) [noise=true , address=77.90.31.19:4855 , piece_id=ITXGC6WJTWODVZC6SH2BNZJOVULHPZEMYUP5IB4ELCVHG6QKKAYA, seg_pos={0 0}, upload=111912]: Done uploading piece. err: ecclient: upload failed (node:19LE2Lq4e6HojnMp9xQ3ySLkRe5PemXTwiQZThN5jT46bQhN6K, address:77.90.31.19:4855): protocol: write tcp 188.166.225.170:53112->77.90.31.19:4855: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:39:05.017061152 (128h10m4.888144187s) [seg_pos={0 0}, upload=111912]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Dec 10 23:39:05.259254672 (128h10m5.130337717s) [upload=111912]: Done uploading segments... Dec 10 23:39:05.259324205 (128h10m5.13040725s) [upload=111912]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 1m0.245953536s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 5.976518597s trace: https://jaeger.dev.storj.io/trace/5b0e6327850e4d93 FAIL (2025-12-10 23:38:52 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.976545579s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 7.525751557s trace: https://jaeger.dev.storj.io/trace/5b0e6327850e4d93 FAIL (2025-12-10 23:38:24 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 7.525780021s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...............................X............................] duration: 4.041317019s trace: https://jaeger.dev.storj.io/trace/30cdf86753ce10d1 FAIL (2025-12-10 23:38:08 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 4.041344801s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 13 out of 60 is failed (threshold=3) [...........................................XXXXX.XXXXX.X..XX] duration: 1m0.246728788s trace: https://jaeger.dev.storj.io/trace/2f4c4bfc30575e34 FAIL (2025-12-10 23:38:04 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== egin Dec 10 23:37:05.496158191 (128h8m5.367241236s) [seg_pos={0 0}, upload=111896]: Begin upload segment... Dec 10 23:37:05.496231721 (128h8m5.367314766s) [seg_pos={0 0}, upload=111896]: Done begin upload segment. Dec 10 23:37:05.496265204 (128h8m5.36734825s) [noise=true , address=77.90.31.18:4810 , piece_id=QCT6VT2NIBWXXA2V6MSOZF76CZABF6F5PGSEMFOR23JWXDIEKSWA, seg_pos={0 0}, upload=111896]: Uploading piece... Dec 10 23:37:05.496275444 (128h8m5.367358489s) [noise=true , address=45.91.116.13:4160 , piece_id=3TLLCRHEO7DFDL6FYXHA7PMUFDR4MRII24LUV4VIGB3YVFWPNO5A, seg_pos={0 0}, upload=111896]: Uploading piece... Dec 10 23:37:05.496838696 (128h8m5.367921741s) [noise=true , address=45.91.116.6:4994 , piece_id=TJCAKJQUX5ZFFQZXNLRGLP2EQ6ZV4CBQKU2HLCS45SKYD57XS2EQ, seg_pos={0 0}, upload=111896]: Uploading piece... Dec 10 23:37:05.497286769 (128h8m5.368369804s) [noise=true , address=145.79.227.44:4209 , piece_id=IQCB7E4BFWHZCFNUIWPECJ4S44TDJC6DS7NZ2PTBFXGXJCFENNWA, seg_pos={0 0}, upload=111896]: Uploading piece... Dec 10 23:37:05.497701689 (128h8m5.368784725s) [noise=true , address=5.231.73.2:4024 , piece_id=FYGOZHHOUTN53AWVTGDBM7YJKIA7FMAT2KV4DPDRB7OACRDOZNNA, seg_pos={0 0}, upload=111896]: Uploading piece... Dec 10 23:37:05.498177706 (128h8m5.369260751s) [noise=true , address=145.79.227.43:4167 , piece_id=6IXA2NV7J2L53YIIKOVQMBHWHFC7NKLLWWMAVAV2534XJCKZ3CFA, seg_pos={0 0}, upload=111896]: Uploading piece... Dec 10 23:37:05.498689680 (128h8m5.369772715s) [noise=true , address=77.90.31.21:4882 , piece_id=Z5SO32UPH2RHCZ3RTF3KEQ77IB5KCYNSLEMNKXJOTE3VYTX7TY5A, seg_pos={0 0}, upload=111896]: Uploading piece... Dec 10 23:37:05.499130761 (128h8m5.370213806s) [noise=true , address=145.79.227.12:4797 , piece_id=BZ5GJUK72L47WGMYA737L7U4PPYXODFOQAF2LB6MAFU7LBS4JRRQ, seg_pos={0 0}, upload=111896]: Uploading piece... Dec 10 23:37:05.499584194 (128h8m5.370667239s) [noise=true , address=77.90.31.19:4859 , piece_id=G4OBDEEKMBKLOKLOY3NMA4P367WTL77DPLV63GF5HWRQ2CQV3YOA, seg_pos={0 0}, upload=111896]: Uploading piece... Dec 10 23:37:05.500038269 (128h8m5.371121314s) [noise=true , address=45.91.116.10:4093 , piece_id=I3JNNM542WMB6JIT6S36SR5SLCDVBYQZSLCF2IUD7HRMWVPTQIRQ, seg_pos={0 0}, upload=111896]: Uploading piece... Dec 10 23:37:05.500520818 (128h8m5.371603863s) [noise=true , address=45.91.116.8:4071 , piece_id=TZR2HIR4VNTL54XHGZITS272SD7JWLXKBW2ZVQ4ODV5KPGORBXEA, seg_pos={0 0}, upload=111896]: Uploading piece... Dec 10 23:37:05.501080273 (128h8m5.372163318s) [noise=true , address=77.90.31.11:4560 , piece_id=DHYUNJFFKH54DRBLIJGJORCZAC6LQ5YJKVNEHTGARLFKUKDP6FMQ, seg_pos={0 0}, upload=111896]: Uploading piece... Dec 10 23:37:05.502616077 (128h8m5.373699122s) [noise=true , address=45.91.116.4:4948 , piece_id=2R7I62QJIHSZAPLY4EKYLJCCG6FXSI724TZY3FWF7DZOCXXFNVKQ, seg_pos={0 0}, upload=111896]: Uploading piece... Dec 10 23:37:05.502909366 (128h8m5.373992411s) [noise=true , address=77.90.31.18:4835 , piece_id=V2S3GXQUA2JT66FZ4Y3XXXJ7PQSALQOVQIH3CTCSHOG2JP742W6A, seg_pos={0 0}, upload=111896]: Uploading piece... Dec 10 23:37:05.504625514 (128h8m5.375708559s) [noise=true , address=77.90.31.15:4723 , piece_id=2EGY5IZLNT5IVYSADI3UHEWEXIEAU2IZWB5K54XL6JDTRKSP35BA, seg_pos={0 0}, upload=111896]: Uploading piece... Dec 10 23:37:05.506415191 (128h8m5.377498236s) [noise=true , address=5.231.73.4:4018 , piece_id=KRDKT2A475AIEXNAEJ5RWSSFZHTZDQKBBP46NKD4GJYISHZFGK3Q, seg_pos={0 0}, upload=111896]: Uploading piece... Dec 10 23:37:05.508089329 (128h8m5.379172374s) [noise=true , address=77.90.31.12:4606 , piece_id=ZWV74JX2DTFRYXKORHYSSBLNTJH6JPRUJ3Z7KGXBEDCHFJY35P3A, seg_pos={0 0}, upload=111896]: Uploading piece... Dec 10 23:37:05.509770570 (128h8m5.380853605s) [noise=true , address=64.49.13.139:4470 , piece_id=4QZUWOUTO56BVOG4ILZW3ZIVDDKNOFJRPC5Z3SUOLRCMCQFFXNHA, seg_pos={0 0}, upload=111896]: Uploading piece... Dec 10 23:37:06.853991857 (128h8m6.725074902s) [upload=111896]: Next returned nil segment Dec 10 23:37:06.854011053 (128h8m6.725094108s) [upload=111896]: Waiting for error group managing segments... Dec 10 23:37:07.192680911 (128h8m7.063763946s) [noise=true , address=64.49.13.139:4470 , piece_id=4QZUWOUTO56BVOG4ILZW3ZIVDDKNOFJRPC5Z3SUOLRCMCQFFXNHA, seg_pos={0 0}, upload=111896]: Done uploading piece. err: <nil> Dec 10 23:37:07.373273807 (128h8m7.244356842s) [noise=true , address=145.79.227.12:4797 , piece_id=BZ5GJUK72L47WGMYA737L7U4PPYXODFOQAF2LB6MAFU7LBS4JRRQ, seg_pos={0 0}, upload=111896]: Done uploading piece. err: <nil> Dec 10 23:37:07.383466474 (128h8m7.254549509s) [noise=true , address=5.231.73.2:4024 , piece_id=FYGOZHHOUTN53AWVTGDBM7YJKIA7FMAT2KV4DPDRB7OACRDOZNNA, seg_pos={0 0}, upload=111896]: Done uploading piece. err: <nil> Dec 10 23:37:07.387817109 (128h8m7.258900144s) [noise=true , address=145.79.227.43:4167 , piece_id=6IXA2NV7J2L53YIIKOVQMBHWHFC7NKLLWWMAVAV2534XJCKZ3CFA, seg_pos={0 0}, upload=111896]: Done uploading piece. err: <nil> Dec 10 23:37:07.390478857 (128h8m7.261561892s) [noise=true , address=5.231.73.4:4018 , piece_id=KRDKT2A475AIEXNAEJ5RWSSFZHTZDQKBBP46NKD4GJYISHZFGK3Q, seg_pos={0 0}, upload=111896]: Done uploading piece. err: <nil> Dec 10 23:37:07.503382663 (128h8m7.374465708s) [noise=true , address=45.91.116.6:4994 , piece_id=TJCAKJQUX5ZFFQZXNLRGLP2EQ6ZV4CBQKU2HLCS45SKYD57XS2EQ, seg_pos={0 0}, upload=111896]: Done uploading piece. err: <nil> Dec 10 23:37:07.547789159 (128h8m7.418872204s) [noise=true , address=45.91.116.4:4948 , piece_id=2R7I62QJIHSZAPLY4EKYLJCCG6FXSI724TZY3FWF7DZOCXXFNVKQ, seg_pos={0 0}, upload=111896]: Done uploading piece. err: <nil> Dec 10 23:37:07.571286772 (128h8m7.442369817s) [noise=true , address=45.91.116.13:4160 , piece_id=3TLLCRHEO7DFDL6FYXHA7PMUFDR4MRII24LUV4VIGB3YVFWPNO5A, seg_pos={0 0}, upload=111896]: Done uploading piece. err: <nil> Dec 10 23:37:07.587561702 (128h8m7.458644737s) [noise=true , address=45.91.116.8:4071 , piece_id=TZR2HIR4VNTL54XHGZITS272SD7JWLXKBW2ZVQ4ODV5KPGORBXEA, seg_pos={0 0}, upload=111896]: Done uploading piece. err: <nil> Dec 10 23:37:07.600339263 (128h8m7.471422298s) [noise=true , address=45.91.116.10:4093 , piece_id=I3JNNM542WMB6JIT6S36SR5SLCDVBYQZSLCF2IUD7HRMWVPTQIRQ, seg_pos={0 0}, upload=111896]: Done uploading piece. err: <nil> Dec 10 23:37:08.301021915 (128h8m8.17210495s) [noise=true , address=145.79.227.44:4209 , piece_id=IQCB7E4BFWHZCFNUIWPECJ4S44TDJC6DS7NZ2PTBFXGXJCFENNWA, seg_pos={0 0}, upload=111896]: Done uploading piece. err: <nil> Dec 10 23:38:04.549847126 (128h9m4.420930172s) [noise=true , address=77.90.31.12:4606 , piece_id=ZWV74JX2DTFRYXKORHYSSBLNTJH6JPRUJ3Z7KGXBEDCHFJY35P3A, seg_pos={0 0}, upload=111896]: Done uploading piece. err: ecclient: upload failed (node:1pLKmTja7ttT6vqjq8vHMYR9RLfnpqxY9xm9GDBG8ez9aHujs, address:77.90.31.12:4606): protocol: rpc: dial tcp 77.90.31.12:4606: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:38:04.549999917 (128h9m4.421082972s) [noise=true , address=77.90.31.18:4810 , piece_id=QCT6VT2NIBWXXA2V6MSOZF76CZABF6F5PGSEMFOR23JWXDIEKSWA, seg_pos={0 0}, upload=111896]: Done uploading piece. err: ecclient: upload failed (node:12mfG3ZeKyFWEPsTgSoR4wqiUYY7YobwmMtsg1GsCorquCZLX5F, address:77.90.31.18:4810): protocol: rpc: dial tcp 77.90.31.18:4810: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:38:04.550117231 (128h9m4.421200276s) [noise=true , address=77.90.31.18:4835 , piece_id=V2S3GXQUA2JT66FZ4Y3XXXJ7PQSALQOVQIH3CTCSHOG2JP742W6A, seg_pos={0 0}, upload=111896]: Done uploading piece. err: ecclient: upload failed (node:12hAQp2Zr7WSaQSTCZf5pwSyuBHHSrZWq5EVz5hFvSA5ddWebqg, address:77.90.31.18:4835): protocol: rpc: dial tcp 77.90.31.18:4835: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:38:04.550224936 (128h9m4.421307981s) [noise=true , address=77.90.31.21:4882 , piece_id=Z5SO32UPH2RHCZ3RTF3KEQ77IB5KCYNSLEMNKXJOTE3VYTX7TY5A, seg_pos={0 0}, upload=111896]: Done uploading piece. err: ecclient: upload failed (node:1qmk23WxPsi88atKkdoQN3vJ3K1YDfNHYBMgYzcc52VURVA9V1, address:77.90.31.21:4882): protocol: rpc: dial tcp 77.90.31.21:4882: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:38:04.550518214 (128h9m4.421601259s) [noise=true , address=77.90.31.19:4859 , piece_id=G4OBDEEKMBKLOKLOY3NMA4P367WTL77DPLV63GF5HWRQ2CQV3YOA, seg_pos={0 0}, upload=111896]: Done uploading piece. err: ecclient: upload failed (node:12Ydy1zHhZCa78uBWuQpgnvbZohVVPaUuaYAqdLpEJQzk423S3h, address:77.90.31.19:4859): protocol: write tcp 188.166.225.170:50480->77.90.31.19:4859: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:38:04.550558050 (128h9m4.421641105s) [noise=true , address=77.90.31.15:4723 , piece_id=2EGY5IZLNT5IVYSADI3UHEWEXIEAU2IZWB5K54XL6JDTRKSP35BA, seg_pos={0 0}, upload=111896]: Done uploading piece. err: ecclient: upload failed (node:1EGKHmd15augxZS76bCbS9f7Kzj8CQSkX1tvSCapydjVqKJseS, address:77.90.31.15:4723): protocol: context deadline exceeded; context deadline exceeded Dec 10 23:38:04.550587045 (128h9m4.42167009s) [noise=true , address=77.90.31.11:4560 , piece_id=DHYUNJFFKH54DRBLIJGJORCZAC6LQ5YJKVNEHTGARLFKUKDP6FMQ, seg_pos={0 0}, upload=111896]: Done uploading piece. err: ecclient: upload failed (node:1YWmyb9dcfFj82ShgsB6EnQXm3mx1YSLvWc6P1SZhREksqXUUH, address:77.90.31.11:4560): protocol: write tcp 188.166.225.170:58370->77.90.31.11:4560: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:38:04.550620028 (128h9m4.421703083s) [seg_pos={0 0}, upload=111896]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Dec 10 23:38:04.794069329 (128h9m4.665152374s) [upload=111896]: Done uploading segments... Dec 10 23:38:04.794110808 (128h9m4.665193853s) [upload=111896]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 1m0.246742673s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 5.763978291s trace: https://jaeger.dev.storj.io/trace/5b0e6327850e4d93 FAIL (2025-12-10 23:37:31 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.763998089s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 6.080814184s trace: https://jaeger.dev.storj.io/trace/5b0e6327850e4d93 FAIL (2025-12-10 23:37:15 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.080858679s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [................................X...........................] duration: 4.140440493s trace: https://jaeger.dev.storj.io/trace/30cdf86753ce10d1 FAIL (2025-12-10 23:37:12 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 4.14048087s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 12 out of 60 is failed (threshold=3) [............................................XXXXX.XXXXX.X..X] duration: 1m0.241635669s trace: https://jaeger.dev.storj.io/trace/155ee94502444de0 FAIL (2025-12-10 23:37:04 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 44587 (128h7m4.644127632s) [upload=111881]: Uploading segments... Dec 10 23:36:04.773375457 (128h7m4.644458502s) [upload=111881]: Got next segment. Inline: false Dec 10 23:36:04.773429930 (128h7m4.644512975s) [upload=111881]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 10 23:36:04.773463093 (128h7m4.644546138s) [upload=111881]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 10 23:36:05.015987325 (128h7m4.887070361s) [seg_pos={0 0}, upload=111881]: Begin upload segment... Dec 10 23:36:05.016073911 (128h7m4.887156956s) [noise=true , address=145.79.227.50:4397 , piece_id=EYC7ZVOTJC3OS375JMBI5IYYBDBUP3CVHO64RNEGVCROIYPHQKQA, seg_pos={0 0}, upload=111881]: Uploading piece... Dec 10 23:36:05.016587990 (128h7m4.887671035s) [seg_pos={0 0}, upload=111881]: Done begin upload segment. Dec 10 23:36:05.016624930 (128h7m4.887707975s) [noise=true , address=5.231.73.3:4014 , piece_id=JZDUE5H7EKR2W75DRYHRTYYXRKY4OQK66GYOPAENRF4LJLMK36WA, seg_pos={0 0}, upload=111881]: Uploading piece... Dec 10 23:36:05.016785045 (128h7m4.88786809s) [noise=true , address=77.90.31.16:4754 , piece_id=FLNG67RB2FI4SKEENZCEH4GY2TXQGPU63GRVB376K6LCT5BVLXPQ, seg_pos={0 0}, upload=111881]: Uploading piece... Dec 10 23:36:05.017163856 (128h7m4.888246901s) [noise=true , address=45.91.116.13:4182 , piece_id=CJRTBOZPDG3NAZRTIUN5MVQJEYSWRHPMBUP3HBGXZCORMVYBPGZQ, seg_pos={0 0}, upload=111881]: Uploading piece... Dec 10 23:36:05.017549069 (128h7m4.888632115s) [noise=true , address=145.79.227.2:4135 , piece_id=7VBZ5ISYPC34HCMFZXV2XGD6TTTLYVQJFTG7INSVDGSQQI76RNEA, seg_pos={0 0}, upload=111881]: Uploading piece... Dec 10 23:36:05.017915357 (128h7m4.888998402s) [noise=true , address=145.79.227.35:4501 , piece_id=P5IPGB4BCDXVI2XZBVQW4KWM3RGS2RQN76RJEIHUECJX5YTXKTMQ, seg_pos={0 0}, upload=111881]: Uploading piece... Dec 10 23:36:05.018349243 (128h7m4.889432288s) [noise=true , address=145.79.227.25:4630 , piece_id=WYZVKCZPZPRNPGH6Z2MGL57TRSAQRC37ZLYBYOR6M3M5LZUXFM6Q, seg_pos={0 0}, upload=111881]: Uploading piece... Dec 10 23:36:05.018408446 (128h7m4.889491491s) [noise=true , address=77.90.31.10:4527 , piece_id=RG72A3I476EXWFRZYHQ5SNSVWGTGWNTBC3IM5PZEFHHF6KA5HDXA, seg_pos={0 0}, upload=111881]: Uploading piece... Dec 10 23:36:05.018688269 (128h7m4.889771314s) [noise=true , address=77.90.31.14:4693 , piece_id=SUM5UD5V7ZA5GLMSXAXLZPLRBO5EQNONGVCAOQ4K2K34WYCPDMUQ, seg_pos={0 0}, upload=111881]: Uploading piece... Dec 10 23:36:05.018855177 (128h7m4.889938222s) [noise=true , address=45.91.116.9:4293 , piece_id=6EDB6HULLLPVVRB52WOYV7LXU2PO64ZOYKRWXWOAOUZPUWFLL4LA, seg_pos={0 0}, upload=111881]: Uploading piece... Dec 10 23:36:05.019881591 (128h7m4.890964636s) [noise=true , address=77.90.31.21:4893 , piece_id=MAZLW3BQ6OWY7LRHEXHHIP7TCD4MR4ABPHJXYJQGH2AU4DFDBF2Q, seg_pos={0 0}, upload=111881]: Uploading piece... Dec 10 23:36:05.020246897 (128h7m4.891330042s) [noise=true , address=5.231.73.4:4008 , piece_id=BTBCAIOUZHLZENR3Q3FZ3KBKGFRB5LWBFYTXTG6RNKNEA7WX72KQ, seg_pos={0 0}, upload=111881]: Uploading piece... Dec 10 23:36:05.020553742 (128h7m4.891636777s) [noise=true , address=45.91.116.7:4040 , piece_id=H4VY5U5GXXE4K4LBHGQUCVTCYTUVJDYBY6QUWSMY2STIXPEJHMMA, seg_pos={0 0}, upload=111881]: Uploading piece... Dec 10 23:36:05.020908176 (128h7m4.891991211s) [noise=true , address=5.231.73.5:4015 , piece_id=NGXNHTQFEG5HGH4VGKDA5JRIDGBPHE22ZX33XUPUSKKJEW3KNJ7A, seg_pos={0 0}, upload=111881]: Uploading piece... Dec 10 23:36:05.021224298 (128h7m4.892307343s) [noise=true , address=145.79.227.46:4625 , piece_id=D2QPUU6KYZOTHXY6XZEARURRHLNQJB5X5PVMGWKRLB32QEJXLRTQ, seg_pos={0 0}, upload=111881]: Uploading piece... Dec 10 23:36:05.021581879 (128h7m4.892664924s) [noise=true , address=77.90.31.11:4561 , piece_id=T3A2B6YX7VTA5WIDMEVJJOH5KCJYECLV3CYUQRCM4SO3WXF2ZOHA, seg_pos={0 0}, upload=111881]: Uploading piece... Dec 10 23:36:05.021981039 (128h7m4.893064084s) [noise=true , address=45.91.116.10:4099 , piece_id=KAZZ6AGADDQJOQPUDQEJK5BKVINMD2NQV6QEX343UQCV3TZA5Q4A, seg_pos={0 0}, upload=111881]: Uploading piece... Dec 10 23:36:05.022359289 (128h7m4.893442334s) [noise=true , address=45.91.116.14:4203 , piece_id=BLVZ5QURSY2BNOHBMU2CO7GMREZX6S7C6LD6KSDP7VBALIKNKZ3Q, seg_pos={0 0}, upload=111881]: Uploading piece... Dec 10 23:36:06.455883142 (128h7m6.326966187s) [upload=111881]: Next returned nil segment Dec 10 23:36:06.455898491 (128h7m6.326981536s) [upload=111881]: Waiting for error group managing segments... Dec 10 23:36:06.819966190 (128h7m6.691049235s) [noise=true , address=5.231.73.5:4015 , piece_id=NGXNHTQFEG5HGH4VGKDA5JRIDGBPHE22ZX33XUPUSKKJEW3KNJ7A, seg_pos={0 0}, upload=111881]: Done uploading piece. err: <nil> Dec 10 23:36:06.861157815 (128h7m6.73224086s) [noise=true , address=5.231.73.4:4008 , piece_id=BTBCAIOUZHLZENR3Q3FZ3KBKGFRB5LWBFYTXTG6RNKNEA7WX72KQ, seg_pos={0 0}, upload=111881]: Done uploading piece. err: <nil> Dec 10 23:36:06.886432242 (128h7m6.757515297s) [noise=true , address=145.79.227.46:4625 , piece_id=D2QPUU6KYZOTHXY6XZEARURRHLNQJB5X5PVMGWKRLB32QEJXLRTQ, seg_pos={0 0}, upload=111881]: Done uploading piece. err: <nil> Dec 10 23:36:06.888722582 (128h7m6.759805627s) [noise=true , address=145.79.227.25:4630 , piece_id=WYZVKCZPZPRNPGH6Z2MGL57TRSAQRC37ZLYBYOR6M3M5LZUXFM6Q, seg_pos={0 0}, upload=111881]: Done uploading piece. err: <nil> Dec 10 23:36:06.904576732 (128h7m6.775659767s) [noise=true , address=145.79.227.2:4135 , piece_id=7VBZ5ISYPC34HCMFZXV2XGD6TTTLYVQJFTG7INSVDGSQQI76RNEA, seg_pos={0 0}, upload=111881]: Done uploading piece. err: <nil> Dec 10 23:36:06.907307141 (128h7m6.778390176s) [noise=true , address=145.79.227.35:4501 , piece_id=P5IPGB4BCDXVI2XZBVQW4KWM3RGS2RQN76RJEIHUECJX5YTXKTMQ, seg_pos={0 0}, upload=111881]: Done uploading piece. err: <nil> Dec 10 23:36:06.917661236 (128h7m6.788744281s) [noise=true , address=5.231.73.3:4014 , piece_id=JZDUE5H7EKR2W75DRYHRTYYXRKY4OQK66GYOPAENRF4LJLMK36WA, seg_pos={0 0}, upload=111881]: Done uploading piece. err: <nil> Dec 10 23:36:07.087204441 (128h7m6.958287486s) [noise=true , address=145.79.227.50:4397 , piece_id=EYC7ZVOTJC3OS375JMBI5IYYBDBUP3CVHO64RNEGVCROIYPHQKQA, seg_pos={0 0}, upload=111881]: Done uploading piece. err: <nil> Dec 10 23:36:07.099327115 (128h7m6.97041015s) [noise=true , address=45.91.116.9:4293 , piece_id=6EDB6HULLLPVVRB52WOYV7LXU2PO64ZOYKRWXWOAOUZPUWFLL4LA, seg_pos={0 0}, upload=111881]: Done uploading piece. err: <nil> Dec 10 23:36:07.102103522 (128h7m6.973186577s) [noise=true , address=45.91.116.13:4182 , piece_id=CJRTBOZPDG3NAZRTIUN5MVQJEYSWRHPMBUP3HBGXZCORMVYBPGZQ, seg_pos={0 0}, upload=111881]: Done uploading piece. err: <nil> Dec 10 23:36:07.118249485 (128h7m6.989332531s) [noise=true , address=45.91.116.10:4099 , piece_id=KAZZ6AGADDQJOQPUDQEJK5BKVINMD2NQV6QEX343UQCV3TZA5Q4A, seg_pos={0 0}, upload=111881]: Done uploading piece. err: <nil> Dec 10 23:36:07.124437038 (128h7m6.995520083s) [noise=true , address=45.91.116.14:4203 , piece_id=BLVZ5QURSY2BNOHBMU2CO7GMREZX6S7C6LD6KSDP7VBALIKNKZ3Q, seg_pos={0 0}, upload=111881]: Done uploading piece. err: <nil> Dec 10 23:36:07.135768264 (128h7m7.006851309s) [noise=true , address=45.91.116.7:4040 , piece_id=H4VY5U5GXXE4K4LBHGQUCVTCYTUVJDYBY6QUWSMY2STIXPEJHMMA, seg_pos={0 0}, upload=111881]: Done uploading piece. err: <nil> Dec 10 23:37:04.099008791 (128h8m3.970091826s) [noise=true , address=77.90.31.16:4754 , piece_id=FLNG67RB2FI4SKEENZCEH4GY2TXQGPU63GRVB376K6LCT5BVLXPQ, seg_pos={0 0}, upload=111881]: Done uploading piece. err: ecclient: upload failed (node:12KKCnEfLWFbLY7ViZ8uzgKfNDeVby4XaoRiBKiQCH39mq82RJ8, address:77.90.31.16:4754): protocol: rpc: dial tcp 77.90.31.16:4754: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:37:04.099110635 (128h8m3.97019368s) [noise=true , address=77.90.31.14:4693 , piece_id=SUM5UD5V7ZA5GLMSXAXLZPLRBO5EQNONGVCAOQ4K2K34WYCPDMUQ, seg_pos={0 0}, upload=111881]: Done uploading piece. err: ecclient: upload failed (node:1RASrnvpAj4ToD4xNvbma1iKZEe2uHK8WosjfZwDDom8Qd17Ey, address:77.90.31.14:4693): protocol: rpc: dial tcp 77.90.31.14:4693: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:37:04.099269117 (128h8m3.970352162s) [noise=true , address=77.90.31.21:4893 , piece_id=MAZLW3BQ6OWY7LRHEXHHIP7TCD4MR4ABPHJXYJQGH2AU4DFDBF2Q, seg_pos={0 0}, upload=111881]: Done uploading piece. err: ecclient: upload failed (node:1NgPjVCyZZAfxVr3WvKrGRjnkpPtUvmQcCMiYntxNKVSWqXcHC, address:77.90.31.21:4893): protocol: rpc: dial tcp 77.90.31.21:4893: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:37:04.099298963 (128h8m3.970382008s) [noise=true , address=77.90.31.10:4527 , piece_id=RG72A3I476EXWFRZYHQ5SNSVWGTGWNTBC3IM5PZEFHHF6KA5HDXA, seg_pos={0 0}, upload=111881]: Done uploading piece. err: ecclient: upload failed (node:1jEyiUUU9NCVFMU8frNggxTTYBMkehsqN4GLsFRkrxubdobGY2, address:77.90.31.10:4527): protocol: write tcp 188.166.225.170:49140->77.90.31.10:4527: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:37:04.099410867 (128h8m3.970493912s) [noise=true , address=77.90.31.11:4561 , piece_id=T3A2B6YX7VTA5WIDMEVJJOH5KCJYECLV3CYUQRCM4SO3WXF2ZOHA, seg_pos={0 0}, upload=111881]: Done uploading piece. err: ecclient: upload failed (node:12bkhuJ5T8SgGFMn6vv6wWnHiJarcMgVuhk6JNmNZoskzwP1gVi, address:77.90.31.11:4561): protocol: write tcp 188.166.225.170:43414->77.90.31.11:4561: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:37:04.099437517 (128h8m3.970520552s) [seg_pos={0 0}, upload=111881]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Dec 10 23:37:04.337865359 (128h8m4.208948414s) [upload=111881]: Done uploading segments... Dec 10 23:37:04.337908802 (128h8m4.208991847s) [upload=111881]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1m0.241668081s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.................................X..........................] duration: 4.099755431s trace: https://jaeger.dev.storj.io/trace/30cdf86753ce10d1 FAIL (2025-12-10 23:36:22 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 4.099783696s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 5.935747335s trace: https://jaeger.dev.storj.io/trace/5b0e6327850e4d93 FAIL (2025-12-10 23:36:07 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.935791279s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 11 out of 60 is failed (threshold=3) [.............................................XXXXX.XXXXX.X..] duration: 6.479566769s trace: https://jaeger.dev.storj.io/trace/1ee4f3dee01cd9d FAIL (2025-12-10 23:35:31 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 128h4m50.163123575s) [upload=111850]: Got next segment. Inline: false Dec 10 23:33:50.292076268 (128h4m50.163159313s) [upload=111850]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 10 23:33:50.292080957 (128h4m50.163164002s) [upload=111850]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 10 23:33:50.539475613 (128h4m50.410558658s) [seg_pos={0 0}, upload=111850]: Begin upload segment... Dec 10 23:33:50.539524155 (128h4m50.4106072s) [seg_pos={0 0}, upload=111850]: Done begin upload segment. Dec 10 23:33:50.539548802 (128h4m50.410631838s) [noise=true , address=64.49.13.137:4503 , piece_id=KQ7WDZITO5YBTZ5UYCUTUDP6FS4KXB5HNTSGWSS5X2IJ5PAGHBXA, seg_pos={0 0}, upload=111850]: Uploading piece... Dec 10 23:33:50.542055025 (128h4m50.41313807s) [noise=true , address=45.91.116.8:4046 , piece_id=KMO4GRXVCAFS4D2CYNYAQKZR3QN7PHX6ZDXD6ISNEDXBX6D2AU5A, seg_pos={0 0}, upload=111850]: Uploading piece... Dec 10 23:33:50.542890536 (128h4m50.413973581s) [noise=true , address=64.49.13.169:4715 , piece_id=MNDNRLDIY6HCYYSSTR4C2PPBK2LKQTWUTGU4HMGCHPQUYG5WN23Q, seg_pos={0 0}, upload=111850]: Uploading piece... Dec 10 23:33:50.543309403 (128h4m50.414392438s) [noise=true , address=77.90.31.19:4865 , piece_id=DWUPW7YHLSD2K5VP7KSISUYWFWI5TDTZR5GRRCNRNGB6BY3ANY2A, seg_pos={0 0}, upload=111850]: Uploading piece... Dec 10 23:33:50.547792670 (128h4m50.418875725s) [noise=true , address=145.79.227.30:4768 , piece_id=HS4LPE3DPT2AIEVJAMLJIQ63LPHXG5UOJJUNZOGSIY7W7OUWC6CQ, seg_pos={0 0}, upload=111850]: Uploading piece... Dec 10 23:33:50.549744968 (128h4m50.420828013s) [noise=true , address=77.90.31.21:4903 , piece_id=UBWQLMNBQF7CZ6DLZVZQ4NKY3LKVU37AVOZKBGLNJMDLC5KEQU3Q, seg_pos={0 0}, upload=111850]: Uploading piece... Dec 10 23:33:50.550840013 (128h4m50.421923048s) [noise=true , address=145.79.227.23:4575 , piece_id=II3UNVY45EPJYTPPN7Z6A25EOSZ6J626WYNVMYNX5OVFA6AZU5GQ, seg_pos={0 0}, upload=111850]: Uploading piece... Dec 10 23:33:50.553417330 (128h4m50.424500375s) [noise=true , address=145.79.227.22:4524 , piece_id=OP2ZOCHDC7VXZLWTZQUCB7EVSZKOSO6MVTD5GNRUYPKCUWG2DD3Q, seg_pos={0 0}, upload=111850]: Uploading piece... Dec 10 23:33:50.553694197 (128h4m50.424777242s) [noise=true , address=45.91.116.7:4018 , piece_id=BJBF5CHMSFEFZNI67XG5XGVC3RGRYVHTLZ4STSU363EVZV5TRW4A, seg_pos={0 0}, upload=111850]: Uploading piece... Dec 10 23:33:50.555409022 (128h4m50.426492068s) [noise=true , address=145.79.227.29:4966 , piece_id=YBOS2MUO4JUDQ3HV7M2AOUNI56Y426SEUKCSD5JQQVIWV3QBC7RQ, seg_pos={0 0}, upload=111850]: Uploading piece... Dec 10 23:33:50.555512289 (128h4m50.426595324s) [noise=true , address=145.79.227.21:4506 , piece_id=X4YZX5PAWFO6F7N2WBYGVPL5OTTOWVJYCG2AIWJAEWYQFT72ZO5Q, seg_pos={0 0}, upload=111850]: Uploading piece... Dec 10 23:33:50.556532381 (128h4m50.427615426s) [noise=true , address=45.91.116.4:4942 , piece_id=VW64X4QLS5ZNEQEK3GROZER3XZNXT7H5NRLBGE3FPEW2ITLWEA6A, seg_pos={0 0}, upload=111850]: Uploading piece... Dec 10 23:33:50.556878220 (128h4m50.427961265s) [noise=true , address=77.90.31.12:4597 , piece_id=BDPROG4D4Q5BAGV6QE7G4RSKOSAOJQTZQTEK4B4A65FHDV2SV7JA, seg_pos={0 0}, upload=111850]: Uploading piece... Dec 10 23:33:50.561372077 (128h4m50.432455122s) [noise=true , address=45.91.116.13:4163 , piece_id=Y3LAFNRWMKM43HCVNDZ2COYUHJ4GOWNZEX5DRHNEVW3OC7BYOULA, seg_pos={0 0}, upload=111850]: Uploading piece... Dec 10 23:33:50.562378764 (128h4m50.433461819s) [noise=true , address=77.90.31.10:4548 , piece_id=QZHYKIUIMAENECRVQC3XMGD3RK2XWY4IMNHMMLMJKTFBBWB74TIQ, seg_pos={0 0}, upload=111850]: Uploading piece... Dec 10 23:33:50.565414925 (128h4m50.43649796s) [noise=true , address=45.91.116.6:4006 , piece_id=ZVRJ2XQE7POLTVLNLIEJFH3W2ETLPLDK2JE4IQZAACZF5GWJQYMA, seg_pos={0 0}, upload=111850]: Uploading piece... Dec 10 23:33:50.566697798 (128h4m50.437780843s) [noise=true , address=145.79.227.16:4923 , piece_id=45VSM2QZZKO4TZN2M6EU37EGBPGLPFBYQ7BIJER5NLMS7FRYXZYQ, seg_pos={0 0}, upload=111850]: Uploading piece... Dec 10 23:33:50.567057984 (128h4m50.438141019s) [noise=true , address=77.90.31.18:4830 , piece_id=XHEJVDXN2O5TWXMEEUU4ZK25L5C3T2VCGGLZEXJQNRRJD54WLIZA, seg_pos={0 0}, upload=111850]: Uploading piece... Dec 10 23:33:51.816967045 (128h4m51.68805008s) [upload=111850]: Next returned nil segment Dec 10 23:33:51.816981353 (128h4m51.688064398s) [upload=111850]: Waiting for error group managing segments... Dec 10 23:33:52.270846940 (128h4m52.141929986s) [noise=true , address=64.49.13.137:4503 , piece_id=KQ7WDZITO5YBTZ5UYCUTUDP6FS4KXB5HNTSGWSS5X2IJ5PAGHBXA, seg_pos={0 0}, upload=111850]: Done uploading piece. err: <nil> Dec 10 23:33:52.412528524 (128h4m52.283611569s) [noise=true , address=145.79.227.22:4524 , piece_id=OP2ZOCHDC7VXZLWTZQUCB7EVSZKOSO6MVTD5GNRUYPKCUWG2DD3Q, seg_pos={0 0}, upload=111850]: Done uploading piece. err: <nil> Dec 10 23:33:52.421951646 (128h4m52.293034691s) [noise=true , address=145.79.227.21:4506 , piece_id=X4YZX5PAWFO6F7N2WBYGVPL5OTTOWVJYCG2AIWJAEWYQFT72ZO5Q, seg_pos={0 0}, upload=111850]: Done uploading piece. err: <nil> Dec 10 23:33:52.437154444 (128h4m52.308237489s) [noise=true , address=145.79.227.23:4575 , piece_id=II3UNVY45EPJYTPPN7Z6A25EOSZ6J626WYNVMYNX5OVFA6AZU5GQ, seg_pos={0 0}, upload=111850]: Done uploading piece. err: <nil> Dec 10 23:33:52.503535424 (128h4m52.374618469s) [noise=true , address=145.79.227.29:4966 , piece_id=YBOS2MUO4JUDQ3HV7M2AOUNI56Y426SEUKCSD5JQQVIWV3QBC7RQ, seg_pos={0 0}, upload=111850]: Done uploading piece. err: <nil> Dec 10 23:33:52.512160257 (128h4m52.383243302s) [noise=true , address=45.91.116.4:4942 , piece_id=VW64X4QLS5ZNEQEK3GROZER3XZNXT7H5NRLBGE3FPEW2ITLWEA6A, seg_pos={0 0}, upload=111850]: Done uploading piece. err: <nil> Dec 10 23:33:52.584211955 (128h4m52.45529502s) [noise=true , address=45.91.116.6:4006 , piece_id=ZVRJ2XQE7POLTVLNLIEJFH3W2ETLPLDK2JE4IQZAACZF5GWJQYMA, seg_pos={0 0}, upload=111850]: Done uploading piece. err: <nil> Dec 10 23:33:52.619426638 (128h4m52.490509683s) [noise=true , address=45.91.116.7:4018 , piece_id=BJBF5CHMSFEFZNI67XG5XGVC3RGRYVHTLZ4STSU363EVZV5TRW4A, seg_pos={0 0}, upload=111850]: Done uploading piece. err: <nil> Dec 10 23:33:52.631602533 (128h4m52.502685568s) [noise=true , address=45.91.116.13:4163 , piece_id=Y3LAFNRWMKM43HCVNDZ2COYUHJ4GOWNZEX5DRHNEVW3OC7BYOULA, seg_pos={0 0}, upload=111850]: Done uploading piece. err: <nil> Dec 10 23:33:52.639163861 (128h4m52.510246906s) [noise=true , address=45.91.116.8:4046 , piece_id=KMO4GRXVCAFS4D2CYNYAQKZR3QN7PHX6ZDXD6ISNEDXBX6D2AU5A, seg_pos={0 0}, upload=111850]: Done uploading piece. err: <nil> Dec 10 23:33:53.420246418 (128h4m53.291329453s) [noise=true , address=145.79.227.30:4768 , piece_id=HS4LPE3DPT2AIEVJAMLJIQ63LPHXG5UOJJUNZOGSIY7W7OUWC6CQ, seg_pos={0 0}, upload=111850]: Done uploading piece. err: <nil> Dec 10 23:33:53.457384064 (128h4m53.328467109s) [noise=true , address=145.79.227.16:4923 , piece_id=45VSM2QZZKO4TZN2M6EU37EGBPGLPFBYQ7BIJER5NLMS7FRYXZYQ, seg_pos={0 0}, upload=111850]: Done uploading piece. err: <nil> Dec 10 23:34:06.412753294 (128h5m6.283836339s) [noise=true , address=64.49.13.169:4715 , piece_id=MNDNRLDIY6HCYYSSTR4C2PPBK2LKQTWUTGU4HMGCHPQUYG5WN23Q, seg_pos={0 0}, upload=111850]: Done uploading piece. err: <nil> Dec 10 23:34:49.595537957 (128h5m49.466620992s) [noise=true , address=77.90.31.18:4830 , piece_id=XHEJVDXN2O5TWXMEEUU4ZK25L5C3T2VCGGLZEXJQNRRJD54WLIZA, seg_pos={0 0}, upload=111850]: Done uploading piece. err: ecclient: upload failed (node:12U2TBUpmxK1xeXXWHv1VNE8xgjQGU3MrcKsEmDRtPvpj2Tnddz, address:77.90.31.18:4830): protocol: write tcp 188.166.225.170:41758->77.90.31.18:4830: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:34:49.595791169 (128h5m49.466874204s) [noise=true , address=77.90.31.10:4548 , piece_id=QZHYKIUIMAENECRVQC3XMGD3RK2XWY4IMNHMMLMJKTFBBWB74TIQ, seg_pos={0 0}, upload=111850]: Done uploading piece. err: ecclient: upload failed (node:1pj8fDt6E3oe2jUTDBrfN3wL7KVuJymqmGf5Y1x4tozmGfTcr7, address:77.90.31.10:4548): protocol: write tcp 188.166.225.170:59690->77.90.31.10:4548: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:34:49.595920916 (128h5m49.467003951s) [noise=true , address=77.90.31.21:4903 , piece_id=UBWQLMNBQF7CZ6DLZVZQ4NKY3LKVU37AVOZKBGLNJMDLC5KEQU3Q, seg_pos={0 0}, upload=111850]: Done uploading piece. err: ecclient: upload failed (node:1rZYWQ1a2sTKnFpF28iQVrem2s8nSpXkeJzZDaS7jvTZNANdfp, address:77.90.31.21:4903): protocol: rpc: dial tcp 77.90.31.21:4903: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:34:49.596044542 (128h5m49.467127587s) [noise=true , address=77.90.31.19:4865 , piece_id=DWUPW7YHLSD2K5VP7KSISUYWFWI5TDTZR5GRRCNRNGB6BY3ANY2A, seg_pos={0 0}, upload=111850]: Done uploading piece. err: ecclient: upload failed (node:1bguBghQonV5sTHaWteyUEnpp5KNzBKbvBpbvrUNFDCveHN77X, address:77.90.31.19:4865): protocol: write tcp 188.166.225.170:59716->77.90.31.19:4865: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:34:49.596216159 (128h5m49.467299194s) [noise=true , address=77.90.31.12:4597 , piece_id=BDPROG4D4Q5BAGV6QE7G4RSKOSAOJQTZQTEK4B4A65FHDV2SV7JA, seg_pos={0 0}, upload=111850]: Done uploading piece. err: ecclient: upload failed (node:127QMaS39fDFZixRP6gVMQkZMefSxLo3ToPctZuDmaARY6nB3H9, address:77.90.31.12:4597): protocol: rpc: dial tcp 77.90.31.12:4597: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Dec 10 23:34:49.596252328 (128h5m49.467335373s) [seg_pos={0 0}, upload=111850]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Dec 10 23:34:49.838751973 (128h5m49.709835008s) [upload=111850]: Done uploading segments... Dec 10 23:34:49.838788051 (128h5m49.709871096s) [upload=111850]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 6.47959879s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..................................X.........................] duration: 3.986729363s trace: https://jaeger.dev.storj.io/trace/30cdf86753ce10d1 FAIL (2025-12-10 23:35:15 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 3.986750593s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 5.812726014s trace: https://jaeger.dev.storj.io/trace/5b0e6327850e4d93 FAIL (2025-12-10 23:35:13 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.812770688s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 5.947281634s trace: https://jaeger.dev.storj.io/trace/5b0e6327850e4d93 FAIL (2025-12-10 23:34:58 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.947290963s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 11 out of 60 is failed (threshold=3) [..............................................XXXXX.XXXXX.X.] duration: 6.660894598s trace: https://jaeger.dev.storj.io/trace/1ee4f3dee01cd9d FAIL (2025-12-10 23:34:56 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 128h4m50.163123575s) [upload=111850]: Got next segment. Inline: false Dec 10 23:33:50.292076268 (128h4m50.163159313s) [upload=111850]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 10 23:33:50.292080957 (128h4m50.163164002s) [upload=111850]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 10 23:33:50.539475613 (128h4m50.410558658s) [seg_pos={0 0}, upload=111850]: Begin upload segment... Dec 10 23:33:50.539524155 (128h4m50.4106072s) [seg_pos={0 0}, upload=111850]: Done begin upload segment. Dec 10 23:33:50.539548802 (128h4m50.410631838s) [noise=true , address=64.49.13.137:4503 , piece_id=KQ7WDZITO5YBTZ5UYCUTUDP6FS4KXB5HNTSGWSS5X2IJ5PAGHBXA, seg_pos={0 0}, upload=111850]: Uploading piece... Dec 10 23:33:50.542055025 (128h4m50.41313807s) [noise=true , address=45.91.116.8:4046 , piece_id=KMO4GRXVCAFS4D2CYNYAQKZR3QN7PHX6ZDXD6ISNEDXBX6D2AU5A, seg_pos={0 0}, upload=111850]: Uploading piece... Dec 10 23:33:50.542890536 (128h4m50.413973581s) [noise=true , address=64.49.13.169:4715 , piece_id=MNDNRLDIY6HCYYSSTR4C2PPBK2LKQTWUTGU4HMGCHPQUYG5WN23Q, seg_pos={0 0}, upload=111850]: Uploading piece... Dec 10 23:33:50.543309403 (128h4m50.414392438s) [noise=true , address=77.90.31.19:4865 , piece_id=DWUPW7YHLSD2K5VP7KSISUYWFWI5TDTZR5GRRCNRNGB6BY3ANY2A, seg_pos={0 0}, upload=111850]: Uploading piece... Dec 10 23:33:50.547792670 (128h4m50.418875725s) [noise=true , address=145.79.227.30:4768 , piece_id=HS4LPE3DPT2AIEVJAMLJIQ63LPHXG5UOJJUNZOGSIY7W7OUWC6CQ, seg_pos={0 0}, upload=111850]: Uploading piece... Dec 10 23:33:50.549744968 (128h4m50.420828013s) [noise=true , address=77.90.31.21:4903 , piece_id=UBWQLMNBQF7CZ6DLZVZQ4NKY3LKVU37AVOZKBGLNJMDLC5KEQU3Q, seg_pos={0 0}, upload=111850]: Uploading piece... Dec 10 23:33:50.550840013 (128h4m50.421923048s) [noise=true , address=145.79.227.23:4575 , piece_id=II3UNVY45EPJYTPPN7Z6A25EOSZ6J626WYNVMYNX5OVFA6AZU5GQ, seg_pos={0 0}, upload=111850]: Uploading piece... Dec 10 23:33:50.553417330 (128h4m50.424500375s) [noise=true , address=145.79.227.22:4524 , piece_id=OP2ZOCHDC7VXZLWTZQUCB7EVSZKOSO6MVTD5GNRUYPKCUWG2DD3Q, seg_pos={0 0}, upload=111850]: Uploading piece... Dec 10 23:33:50.553694197 (128h4m50.424777242s) [noise=true , address=45.91.116.7:4018 , piece_id=BJBF5CHMSFEFZNI67XG5XGVC3RGRYVHTLZ4STSU363EVZV5TRW4A, seg_pos={0 0}, upload=111850]: Uploading piece... Dec 10 23:33:50.555409022 (128h4m50.426492068s) [noise=true , address=145.79.227.29:4966 , piece_id=YBOS2MUO4JUDQ3HV7M2AOUNI56Y426SEUKCSD5JQQVIWV3QBC7RQ, seg_pos={0 0}, upload=111850]: Uploading piece... Dec 10 23:33:50.555512289 (128h4m50.426595324s) [noise=true , address=145.79.227.21:4506 , piece_id=X4YZX5PAWFO6F7N2WBYGVPL5OTTOWVJYCG2AIWJAEWYQFT72ZO5Q, seg_pos={0 0}, upload=111850]: Uploading piece... Dec 10 23:33:50.556532381 (128h4m50.427615426s) [noise=true , address=45.91.116.4:4942 , piece_id=VW64X4QLS5ZNEQEK3GROZER3XZNXT7H5NRLBGE3FPEW2ITLWEA6A, seg_pos={0 0}, upload=111850]: Uploading piece... Dec 10 23:33:50.556878220 (128h4m50.427961265s) [noise=true , address=77.90.31.12:4597 , piece_id=BDPROG4D4Q5BAGV6QE7G4RSKOSAOJQTZQTEK4B4A65FHDV2SV7JA, seg_pos={0 0}, upload=111850]: Uploading piece... Dec 10 23:33:50.561372077 (128h4m50.432455122s) [noise=true , address=45.91.116.13:4163 , piece_id=Y3LAFNRWMKM43HCVNDZ2COYUHJ4GOWNZEX5DRHNEVW3OC7BYOULA, seg_pos={0 0}, upload=111850]: Uploading piece... Dec 10 23:33:50.562378764 (128h4m50.433461819s) [noise=true , address=77.90.31.10:4548 , piece_id=QZHYKIUIMAENECRVQC3XMGD3RK2XWY4IMNHMMLMJKTFBBWB74TIQ, seg_pos={0 0}, upload=111850]: Uploading piece... Dec 10 23:33:50.565414925 (128h4m50.43649796s) [noise=true , address=45.91.116.6:4006 , piece_id=ZVRJ2XQE7POLTVLNLIEJFH3W2ETLPLDK2JE4IQZAACZF5GWJQYMA, seg_pos={0 0}, upload=111850]: Uploading piece... Dec 10 23:33:50.566697798 (128h4m50.437780843s) [noise=true , address=145.79.227.16:4923 , piece_id=45VSM2QZZKO4TZN2M6EU37EGBPGLPFBYQ7BIJER5NLMS7FRYXZYQ, seg_pos={0 0}, upload=111850]: Uploading piece... Dec 10 23:33:50.567057984 (128h4m50.438141019s) [noise=true , address=77.90.31.18:4830 , piece_id=XHEJVDXN2O5TWXMEEUU4ZK25L5C3T2VCGGLZEXJQNRRJD54WLIZA, seg_pos={0 0}, upload=111850]: Uploading piece... Dec 10 23:33:51.816967045 (128h4m51.68805008s) [upload=111850]: Next returned nil segment Dec 10 23:33:51.816981353 (128h4m51.688064398s) [upload=111850]: Waiting for error group managing segments... Dec 10 23:33:52.270846940 (128h4m52.141929986s) [noise=true , address=64.49.13.137:4503 , piece_id=KQ7WDZITO5YBTZ5UYCUTUDP6FS4KXB5HNTSGWSS5X2IJ5PAGHBXA, seg_pos={0 0}, upload=111850]: Done uploading piece. err: <nil> Dec 10 23:33:52.412528524 (128h4m52.283611569s) [noise=true , address=145.79.227.22:4524 , piece_id=OP2ZOCHDC7VXZLWTZQUCB7EVSZKOSO6MVTD5GNRUYPKCUWG2DD3Q, seg_pos={0 0}, upload=111850]: Done uploading piece. err: <nil> Dec 10 23:33:52.421951646 (128h4m52.293034691s) [noise=true , address=145.79.227.21:4506 , piece_id=X4YZX5PAWFO6F7N2WBYGVPL5OTTOWVJYCG2AIWJAEWYQFT72ZO5Q, seg_pos={0 0}, upload=111850]: Done uploading piece. err: <nil> Dec 10 23:33:52.437154444 (128h4m52.308237489s) [noise=true , address=145.79.227.23:4575 , piece_id=II3UNVY45EPJYTPPN7Z6A25EOSZ6J626WYNVMYNX5OVFA6AZU5GQ, seg_pos={0 0}, upload=111850]: Done uploading piece. err: <nil> Dec 10 23:33:52.503535424 (128h4m52.374618469s) [noise=true , address=145.79.227.29:4966 , piece_id=YBOS2MUO4JUDQ3HV7M2AOUNI56Y426SEUKCSD5JQQVIWV3QBC7RQ, seg_pos={0 0}, upload=111850]: Done uploading piece. err: <nil> Dec 10 23:33:52.512160257 (128h4m52.383243302s) [noise=true , address=45.91.116.4:4942 , piece_id=VW64X4QLS5ZNEQEK3GROZER3XZNXT7H5NRLBGE3FPEW2ITLWEA6A, seg_pos={0 0}, upload=111850]: Done uploading piece. err: <nil> Dec 10 23:33:52.584211955 (128h4m52.45529502s) [noise=true , address=45.91.116.6:4006 , piece_id=ZVRJ2XQE7POLTVLNLIEJFH3W2ETLPLDK2JE4IQZAACZF5GWJQYMA, seg_pos={0 0}, upload=111850]: Done uploading piece. err: <nil> Dec 10 23:33:52.619426638 (128h4m52.490509683s) [noise=true , address=45.91.116.7:4018 , piece_id=BJBF5CHMSFEFZNI67XG5XGVC3RGRYVHTLZ4STSU363EVZV5TRW4A, seg_pos={0 0}, upload=111850]: Done uploading piece. err: <nil> Dec 10 23:33:52.631602533 (128h4m52.502685568s) [noise=true , address=45.91.116.13:4163 , piece_id=Y3LAFNRWMKM43HCVNDZ2COYUHJ4GOWNZEX5DRHNEVW3OC7BYOULA, seg_pos={0 0}, upload=111850]: Done uploading piece. err: <nil> Dec 10 23:33:52.639163861 (128h4m52.510246906s) [noise=true , address=45.91.116.8:4046 , piece_id=KMO4GRXVCAFS4D2CYNYAQKZR3QN7PHX6ZDXD6ISNEDXBX6D2AU5A, seg_pos={0 0}, upload=111850]: Done uploading piece. err: <nil> Dec 10 23:33:53.420246418 (128h4m53.291329453s) [noise=true , address=145.79.227.30:4768 , piece_id=HS4LPE3DPT2AIEVJAMLJIQ63LPHXG5UOJJUNZOGSIY7W7OUWC6CQ, seg_pos={0 0}, upload=111850]: Done uploading piece. err: <nil> Dec 10 23:33:53.457384064 (128h4m53.328467109s) [noise=true , address=145.79.227.16:4923 , piece_id=45VSM2QZZKO4TZN2M6EU37EGBPGLPFBYQ7BIJER5NLMS7FRYXZYQ, seg_pos={0 0}, upload=111850]: Done uploading piece. err: <nil> Dec 10 23:34:06.412753294 (128h5m6.283836339s) [noise=true , address=64.49.13.169:4715 , piece_id=MNDNRLDIY6HCYYSSTR4C2PPBK2LKQTWUTGU4HMGCHPQUYG5WN23Q, seg_pos={0 0}, upload=111850]: Done uploading piece. err: <nil> Dec 10 23:34:49.595537957 (128h5m49.466620992s) [noise=true , address=77.90.31.18:4830 , piece_id=XHEJVDXN2O5TWXMEEUU4ZK25L5C3T2VCGGLZEXJQNRRJD54WLIZA, seg_pos={0 0}, upload=111850]: Done uploading piece. err: ecclient: upload failed (node:12U2TBUpmxK1xeXXWHv1VNE8xgjQGU3MrcKsEmDRtPvpj2Tnddz, address:77.90.31.18:4830): protocol: write tcp 188.166.225.170:41758->77.90.31.18:4830: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:34:49.595791169 (128h5m49.466874204s) [noise=true , address=77.90.31.10:4548 , piece_id=QZHYKIUIMAENECRVQC3XMGD3RK2XWY4IMNHMMLMJKTFBBWB74TIQ, seg_pos={0 0}, upload=111850]: Done uploading piece. err: ecclient: upload failed (node:1pj8fDt6E3oe2jUTDBrfN3wL7KVuJymqmGf5Y1x4tozmGfTcr7, address:77.90.31.10:4548): protocol: write tcp 188.166.225.170:59690->77.90.31.10:4548: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:34:49.595920916 (128h5m49.467003951s) [noise=true , address=77.90.31.21:4903 , piece_id=UBWQLMNBQF7CZ6DLZVZQ4NKY3LKVU37AVOZKBGLNJMDLC5KEQU3Q, seg_pos={0 0}, upload=111850]: Done uploading piece. err: ecclient: upload failed (node:1rZYWQ1a2sTKnFpF28iQVrem2s8nSpXkeJzZDaS7jvTZNANdfp, address:77.90.31.21:4903): protocol: rpc: dial tcp 77.90.31.21:4903: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:34:49.596044542 (128h5m49.467127587s) [noise=true , address=77.90.31.19:4865 , piece_id=DWUPW7YHLSD2K5VP7KSISUYWFWI5TDTZR5GRRCNRNGB6BY3ANY2A, seg_pos={0 0}, upload=111850]: Done uploading piece. err: ecclient: upload failed (node:1bguBghQonV5sTHaWteyUEnpp5KNzBKbvBpbvrUNFDCveHN77X, address:77.90.31.19:4865): protocol: write tcp 188.166.225.170:59716->77.90.31.19:4865: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:34:49.596216159 (128h5m49.467299194s) [noise=true , address=77.90.31.12:4597 , piece_id=BDPROG4D4Q5BAGV6QE7G4RSKOSAOJQTZQTEK4B4A65FHDV2SV7JA, seg_pos={0 0}, upload=111850]: Done uploading piece. err: ecclient: upload failed (node:127QMaS39fDFZixRP6gVMQkZMefSxLo3ToPctZuDmaARY6nB3H9, address:77.90.31.12:4597): protocol: rpc: dial tcp 77.90.31.12:4597: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Dec 10 23:34:49.596252328 (128h5m49.467335373s) [seg_pos={0 0}, upload=111850]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Dec 10 23:34:49.838751973 (128h5m49.709835008s) [upload=111850]: Done uploading segments... Dec 10 23:34:49.838788051 (128h5m49.709871096s) [upload=111850]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 6.660911s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.001504214s trace: https://jaeger.dev.storj.io/trace/5b0e6327850e4d93 FAIL (2025-12-10 23:34:51 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001540652s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 11 out of 60 is failed (threshold=3) [...............................................XXXXX.XXXXX.X] duration: 1m0.245748684s trace: https://jaeger.dev.storj.io/trace/1ee4f3dee01cd9d FAIL (2025-12-10 23:34:49 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 128h4m50.163123575s) [upload=111850]: Got next segment. Inline: false Dec 10 23:33:50.292076268 (128h4m50.163159313s) [upload=111850]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 10 23:33:50.292080957 (128h4m50.163164002s) [upload=111850]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 10 23:33:50.539475613 (128h4m50.410558658s) [seg_pos={0 0}, upload=111850]: Begin upload segment... Dec 10 23:33:50.539524155 (128h4m50.4106072s) [seg_pos={0 0}, upload=111850]: Done begin upload segment. Dec 10 23:33:50.539548802 (128h4m50.410631838s) [noise=true , address=64.49.13.137:4503 , piece_id=KQ7WDZITO5YBTZ5UYCUTUDP6FS4KXB5HNTSGWSS5X2IJ5PAGHBXA, seg_pos={0 0}, upload=111850]: Uploading piece... Dec 10 23:33:50.542055025 (128h4m50.41313807s) [noise=true , address=45.91.116.8:4046 , piece_id=KMO4GRXVCAFS4D2CYNYAQKZR3QN7PHX6ZDXD6ISNEDXBX6D2AU5A, seg_pos={0 0}, upload=111850]: Uploading piece... Dec 10 23:33:50.542890536 (128h4m50.413973581s) [noise=true , address=64.49.13.169:4715 , piece_id=MNDNRLDIY6HCYYSSTR4C2PPBK2LKQTWUTGU4HMGCHPQUYG5WN23Q, seg_pos={0 0}, upload=111850]: Uploading piece... Dec 10 23:33:50.543309403 (128h4m50.414392438s) [noise=true , address=77.90.31.19:4865 , piece_id=DWUPW7YHLSD2K5VP7KSISUYWFWI5TDTZR5GRRCNRNGB6BY3ANY2A, seg_pos={0 0}, upload=111850]: Uploading piece... Dec 10 23:33:50.547792670 (128h4m50.418875725s) [noise=true , address=145.79.227.30:4768 , piece_id=HS4LPE3DPT2AIEVJAMLJIQ63LPHXG5UOJJUNZOGSIY7W7OUWC6CQ, seg_pos={0 0}, upload=111850]: Uploading piece... Dec 10 23:33:50.549744968 (128h4m50.420828013s) [noise=true , address=77.90.31.21:4903 , piece_id=UBWQLMNBQF7CZ6DLZVZQ4NKY3LKVU37AVOZKBGLNJMDLC5KEQU3Q, seg_pos={0 0}, upload=111850]: Uploading piece... Dec 10 23:33:50.550840013 (128h4m50.421923048s) [noise=true , address=145.79.227.23:4575 , piece_id=II3UNVY45EPJYTPPN7Z6A25EOSZ6J626WYNVMYNX5OVFA6AZU5GQ, seg_pos={0 0}, upload=111850]: Uploading piece... Dec 10 23:33:50.553417330 (128h4m50.424500375s) [noise=true , address=145.79.227.22:4524 , piece_id=OP2ZOCHDC7VXZLWTZQUCB7EVSZKOSO6MVTD5GNRUYPKCUWG2DD3Q, seg_pos={0 0}, upload=111850]: Uploading piece... Dec 10 23:33:50.553694197 (128h4m50.424777242s) [noise=true , address=45.91.116.7:4018 , piece_id=BJBF5CHMSFEFZNI67XG5XGVC3RGRYVHTLZ4STSU363EVZV5TRW4A, seg_pos={0 0}, upload=111850]: Uploading piece... Dec 10 23:33:50.555409022 (128h4m50.426492068s) [noise=true , address=145.79.227.29:4966 , piece_id=YBOS2MUO4JUDQ3HV7M2AOUNI56Y426SEUKCSD5JQQVIWV3QBC7RQ, seg_pos={0 0}, upload=111850]: Uploading piece... Dec 10 23:33:50.555512289 (128h4m50.426595324s) [noise=true , address=145.79.227.21:4506 , piece_id=X4YZX5PAWFO6F7N2WBYGVPL5OTTOWVJYCG2AIWJAEWYQFT72ZO5Q, seg_pos={0 0}, upload=111850]: Uploading piece... Dec 10 23:33:50.556532381 (128h4m50.427615426s) [noise=true , address=45.91.116.4:4942 , piece_id=VW64X4QLS5ZNEQEK3GROZER3XZNXT7H5NRLBGE3FPEW2ITLWEA6A, seg_pos={0 0}, upload=111850]: Uploading piece... Dec 10 23:33:50.556878220 (128h4m50.427961265s) [noise=true , address=77.90.31.12:4597 , piece_id=BDPROG4D4Q5BAGV6QE7G4RSKOSAOJQTZQTEK4B4A65FHDV2SV7JA, seg_pos={0 0}, upload=111850]: Uploading piece... Dec 10 23:33:50.561372077 (128h4m50.432455122s) [noise=true , address=45.91.116.13:4163 , piece_id=Y3LAFNRWMKM43HCVNDZ2COYUHJ4GOWNZEX5DRHNEVW3OC7BYOULA, seg_pos={0 0}, upload=111850]: Uploading piece... Dec 10 23:33:50.562378764 (128h4m50.433461819s) [noise=true , address=77.90.31.10:4548 , piece_id=QZHYKIUIMAENECRVQC3XMGD3RK2XWY4IMNHMMLMJKTFBBWB74TIQ, seg_pos={0 0}, upload=111850]: Uploading piece... Dec 10 23:33:50.565414925 (128h4m50.43649796s) [noise=true , address=45.91.116.6:4006 , piece_id=ZVRJ2XQE7POLTVLNLIEJFH3W2ETLPLDK2JE4IQZAACZF5GWJQYMA, seg_pos={0 0}, upload=111850]: Uploading piece... Dec 10 23:33:50.566697798 (128h4m50.437780843s) [noise=true , address=145.79.227.16:4923 , piece_id=45VSM2QZZKO4TZN2M6EU37EGBPGLPFBYQ7BIJER5NLMS7FRYXZYQ, seg_pos={0 0}, upload=111850]: Uploading piece... Dec 10 23:33:50.567057984 (128h4m50.438141019s) [noise=true , address=77.90.31.18:4830 , piece_id=XHEJVDXN2O5TWXMEEUU4ZK25L5C3T2VCGGLZEXJQNRRJD54WLIZA, seg_pos={0 0}, upload=111850]: Uploading piece... Dec 10 23:33:51.816967045 (128h4m51.68805008s) [upload=111850]: Next returned nil segment Dec 10 23:33:51.816981353 (128h4m51.688064398s) [upload=111850]: Waiting for error group managing segments... Dec 10 23:33:52.270846940 (128h4m52.141929986s) [noise=true , address=64.49.13.137:4503 , piece_id=KQ7WDZITO5YBTZ5UYCUTUDP6FS4KXB5HNTSGWSS5X2IJ5PAGHBXA, seg_pos={0 0}, upload=111850]: Done uploading piece. err: <nil> Dec 10 23:33:52.412528524 (128h4m52.283611569s) [noise=true , address=145.79.227.22:4524 , piece_id=OP2ZOCHDC7VXZLWTZQUCB7EVSZKOSO6MVTD5GNRUYPKCUWG2DD3Q, seg_pos={0 0}, upload=111850]: Done uploading piece. err: <nil> Dec 10 23:33:52.421951646 (128h4m52.293034691s) [noise=true , address=145.79.227.21:4506 , piece_id=X4YZX5PAWFO6F7N2WBYGVPL5OTTOWVJYCG2AIWJAEWYQFT72ZO5Q, seg_pos={0 0}, upload=111850]: Done uploading piece. err: <nil> Dec 10 23:33:52.437154444 (128h4m52.308237489s) [noise=true , address=145.79.227.23:4575 , piece_id=II3UNVY45EPJYTPPN7Z6A25EOSZ6J626WYNVMYNX5OVFA6AZU5GQ, seg_pos={0 0}, upload=111850]: Done uploading piece. err: <nil> Dec 10 23:33:52.503535424 (128h4m52.374618469s) [noise=true , address=145.79.227.29:4966 , piece_id=YBOS2MUO4JUDQ3HV7M2AOUNI56Y426SEUKCSD5JQQVIWV3QBC7RQ, seg_pos={0 0}, upload=111850]: Done uploading piece. err: <nil> Dec 10 23:33:52.512160257 (128h4m52.383243302s) [noise=true , address=45.91.116.4:4942 , piece_id=VW64X4QLS5ZNEQEK3GROZER3XZNXT7H5NRLBGE3FPEW2ITLWEA6A, seg_pos={0 0}, upload=111850]: Done uploading piece. err: <nil> Dec 10 23:33:52.584211955 (128h4m52.45529502s) [noise=true , address=45.91.116.6:4006 , piece_id=ZVRJ2XQE7POLTVLNLIEJFH3W2ETLPLDK2JE4IQZAACZF5GWJQYMA, seg_pos={0 0}, upload=111850]: Done uploading piece. err: <nil> Dec 10 23:33:52.619426638 (128h4m52.490509683s) [noise=true , address=45.91.116.7:4018 , piece_id=BJBF5CHMSFEFZNI67XG5XGVC3RGRYVHTLZ4STSU363EVZV5TRW4A, seg_pos={0 0}, upload=111850]: Done uploading piece. err: <nil> Dec 10 23:33:52.631602533 (128h4m52.502685568s) [noise=true , address=45.91.116.13:4163 , piece_id=Y3LAFNRWMKM43HCVNDZ2COYUHJ4GOWNZEX5DRHNEVW3OC7BYOULA, seg_pos={0 0}, upload=111850]: Done uploading piece. err: <nil> Dec 10 23:33:52.639163861 (128h4m52.510246906s) [noise=true , address=45.91.116.8:4046 , piece_id=KMO4GRXVCAFS4D2CYNYAQKZR3QN7PHX6ZDXD6ISNEDXBX6D2AU5A, seg_pos={0 0}, upload=111850]: Done uploading piece. err: <nil> Dec 10 23:33:53.420246418 (128h4m53.291329453s) [noise=true , address=145.79.227.30:4768 , piece_id=HS4LPE3DPT2AIEVJAMLJIQ63LPHXG5UOJJUNZOGSIY7W7OUWC6CQ, seg_pos={0 0}, upload=111850]: Done uploading piece. err: <nil> Dec 10 23:33:53.457384064 (128h4m53.328467109s) [noise=true , address=145.79.227.16:4923 , piece_id=45VSM2QZZKO4TZN2M6EU37EGBPGLPFBYQ7BIJER5NLMS7FRYXZYQ, seg_pos={0 0}, upload=111850]: Done uploading piece. err: <nil> Dec 10 23:34:06.412753294 (128h5m6.283836339s) [noise=true , address=64.49.13.169:4715 , piece_id=MNDNRLDIY6HCYYSSTR4C2PPBK2LKQTWUTGU4HMGCHPQUYG5WN23Q, seg_pos={0 0}, upload=111850]: Done uploading piece. err: <nil> Dec 10 23:34:49.595537957 (128h5m49.466620992s) [noise=true , address=77.90.31.18:4830 , piece_id=XHEJVDXN2O5TWXMEEUU4ZK25L5C3T2VCGGLZEXJQNRRJD54WLIZA, seg_pos={0 0}, upload=111850]: Done uploading piece. err: ecclient: upload failed (node:12U2TBUpmxK1xeXXWHv1VNE8xgjQGU3MrcKsEmDRtPvpj2Tnddz, address:77.90.31.18:4830): protocol: write tcp 188.166.225.170:41758->77.90.31.18:4830: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:34:49.595791169 (128h5m49.466874204s) [noise=true , address=77.90.31.10:4548 , piece_id=QZHYKIUIMAENECRVQC3XMGD3RK2XWY4IMNHMMLMJKTFBBWB74TIQ, seg_pos={0 0}, upload=111850]: Done uploading piece. err: ecclient: upload failed (node:1pj8fDt6E3oe2jUTDBrfN3wL7KVuJymqmGf5Y1x4tozmGfTcr7, address:77.90.31.10:4548): protocol: write tcp 188.166.225.170:59690->77.90.31.10:4548: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:34:49.595920916 (128h5m49.467003951s) [noise=true , address=77.90.31.21:4903 , piece_id=UBWQLMNBQF7CZ6DLZVZQ4NKY3LKVU37AVOZKBGLNJMDLC5KEQU3Q, seg_pos={0 0}, upload=111850]: Done uploading piece. err: ecclient: upload failed (node:1rZYWQ1a2sTKnFpF28iQVrem2s8nSpXkeJzZDaS7jvTZNANdfp, address:77.90.31.21:4903): protocol: rpc: dial tcp 77.90.31.21:4903: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:34:49.596044542 (128h5m49.467127587s) [noise=true , address=77.90.31.19:4865 , piece_id=DWUPW7YHLSD2K5VP7KSISUYWFWI5TDTZR5GRRCNRNGB6BY3ANY2A, seg_pos={0 0}, upload=111850]: Done uploading piece. err: ecclient: upload failed (node:1bguBghQonV5sTHaWteyUEnpp5KNzBKbvBpbvrUNFDCveHN77X, address:77.90.31.19:4865): protocol: write tcp 188.166.225.170:59716->77.90.31.19:4865: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:34:49.596216159 (128h5m49.467299194s) [noise=true , address=77.90.31.12:4597 , piece_id=BDPROG4D4Q5BAGV6QE7G4RSKOSAOJQTZQTEK4B4A65FHDV2SV7JA, seg_pos={0 0}, upload=111850]: Done uploading piece. err: ecclient: upload failed (node:127QMaS39fDFZixRP6gVMQkZMefSxLo3ToPctZuDmaARY6nB3H9, address:77.90.31.12:4597): protocol: rpc: dial tcp 77.90.31.12:4597: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Dec 10 23:34:49.596252328 (128h5m49.467335373s) [seg_pos={0 0}, upload=111850]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Dec 10 23:34:49.838751973 (128h5m49.709835008s) [upload=111850]: Done uploading segments... Dec 10 23:34:49.838788051 (128h5m49.709871096s) [upload=111850]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1m0.24577309s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...................................X........................] duration: 4.105205794s trace: https://jaeger.dev.storj.io/trace/30cdf86753ce10d1 FAIL (2025-12-10 23:34:03 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 4.105243556s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 10 out of 60 is failed (threshold=3) [................................................XXXXX.XXXXX.] duration: 6.433608106s trace: https://jaeger.dev.storj.io/trace/5b106df65b5519b FAIL (2025-12-10 23:33:34 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== pload=111828]: Begin upload segment... Dec 10 23:32:24.390786338 (128h3m24.261869383s) [seg_pos={0 0}, upload=111828]: Done begin upload segment. Dec 10 23:32:24.390809502 (128h3m24.261892537s) [noise=true , address=145.79.227.9:4382 , piece_id=GFQZFT6GWRBLCNKCDIO26QNGW5TLPPUEFZLZ227UVW7O5KHV6CGQ, seg_pos={0 0}, upload=111828]: Uploading piece... Dec 10 23:32:24.390930392 (128h3m24.262013427s) [noise=true , address=77.90.31.16:4749 , piece_id=OP4KBGPYLUCOVLXPKMSP2A2TMR2WU5KZUKNBLGEEITVP5GCLGZNA, seg_pos={0 0}, upload=111828]: Uploading piece... Dec 10 23:32:24.391622149 (128h3m24.262705185s) [noise=true , address=77.90.31.18:4838 , piece_id=MR3RYVKHQSNQDWEO5KJCBBH6CQKUFZELT5IUKH3RRLCOGOUN77KA, seg_pos={0 0}, upload=111828]: Uploading piece... Dec 10 23:32:24.391938291 (128h3m24.263021326s) [noise=true , address=77.90.31.15:4728 , piece_id=5S3AZ2ASGL4W4O5M243GYHVH6SNJVXNCGWURL237OZOKYV4PIV4Q, seg_pos={0 0}, upload=111828]: Uploading piece... Dec 10 23:32:24.392227722 (128h3m24.263310757s) [noise=true , address=145.79.227.49:4365 , piece_id=3CDJOGWEAZJ6OWKS2ZTOX33CN7VU76423B2WU5F4RWNIZDMYANSA, seg_pos={0 0}, upload=111828]: Uploading piece... Dec 10 23:32:24.392239205 (128h3m24.26332224s) [noise=true , address=45.91.116.6:4990 , piece_id=SENJU357NGTDHZNWOCH2AUNNWKVV2DUYR3SDO5GNDDRZQ3XEHLXQ, seg_pos={0 0}, upload=111828]: Uploading piece... Dec 10 23:32:24.392594541 (128h3m24.263677586s) [noise=true , address=77.90.31.15:4715 , piece_id=GSNDMPPZ6YXF2A3CORCGSIAJXX5UB5OSHRZGWZTKLMYVNRDS4HWA, seg_pos={0 0}, upload=111828]: Uploading piece... Dec 10 23:32:24.392661579 (128h3m24.263744624s) [noise=true , address=145.79.227.47:4268 , piece_id=K5BTBN3PS4K4WI77ER4BL73B6XB4SJOJNTSSFFCWM7FUJR2ESYNQ, seg_pos={0 0}, upload=111828]: Uploading piece... Dec 10 23:32:24.392952142 (128h3m24.264035197s) [noise=true , address=45.91.116.5:4322 , piece_id=3CKTYVMA2XG7LJWNF6E26QFR3RQQFQKMNYAWVKOERQOLMMN5X43A, seg_pos={0 0}, upload=111828]: Uploading piece... Dec 10 23:32:24.392981077 (128h3m24.264064112s) [noise=true , address=145.79.227.41:4118 , piece_id=7SIIFPT7FDYBJGU7BKNJAH7QL5ACZTXIE6EAZ35X2Q473M36TRAA, seg_pos={0 0}, upload=111828]: Uploading piece... Dec 10 23:32:24.393296888 (128h3m24.264379923s) [noise=true , address=145.79.227.25:4628 , piece_id=6WZWP6O4Z6DVL4MV4OJVC5FEHJ3AKKBTDWUI3BZGKL674JNMXKTA, seg_pos={0 0}, upload=111828]: Uploading piece... Dec 10 23:32:24.393314242 (128h3m24.264397277s) [noise=true , address=77.90.31.11:4564 , piece_id=BYIG5GF6AOBHKE6WYOGOTGHKP5HDYFNDBVIYHU5MNYEMCZVMO4HQ, seg_pos={0 0}, upload=111828]: Uploading piece... Dec 10 23:32:24.393584496 (128h3m24.264667541s) [noise=true , address=77.90.31.13:4630 , piece_id=6VPE544CN6FNHTS7GYOBN6MO7CCR4T7EEPXL2LB3HMD6H5DIOTNQ, seg_pos={0 0}, upload=111828]: Uploading piece... Dec 10 23:32:24.393648378 (128h3m24.264731423s) [noise=true , address=45.91.116.4:4963 , piece_id=DP4IKIY3BRDEGMCV3LHSUQHBF2WMGENSQ5OMYUZDX3Z65JAO564Q, seg_pos={0 0}, upload=111828]: Uploading piece... Dec 10 23:32:24.394019685 (128h3m24.26510272s) [noise=true , address=45.91.116.12:4246 , piece_id=JPQNIW7L776ZZZSDJWI2AHMMHQ57LHG6MQXMXXKIWRC7M64TP3RA, seg_pos={0 0}, upload=111828]: Uploading piece... Dec 10 23:32:24.394444433 (128h3m24.265527479s) [noise=true , address=77.90.31.19:4859 , piece_id=GDDT7KP34HDFEDMVHETUOAK7KZFY2Z3UPZX4B2WONBCDJE66TITA, seg_pos={0 0}, upload=111828]: Uploading piece... Dec 10 23:32:24.394933835 (128h3m24.26601689s) [noise=true , address=77.90.31.17:4780 , piece_id=263Z64K3MMOS27TK5MO33MSS5PBBLZ6UJGV44LNGN4TDB4YQDYGQ, seg_pos={0 0}, upload=111828]: Uploading piece... Dec 10 23:32:24.397086755 (128h3m24.26816979s) [noise=true , address=145.79.227.7:4296 , piece_id=ANDIF4JW2GAUYO6L45QBMEAOC36T77FOENXMM2C6DPNEPABU4BPQ, seg_pos={0 0}, upload=111828]: Uploading piece... Dec 10 23:32:25.861307488 (128h3m25.732390533s) [upload=111828]: Next returned nil segment Dec 10 23:32:25.861325311 (128h3m25.732408356s) [upload=111828]: Waiting for error group managing segments... Dec 10 23:32:26.249668283 (128h3m26.120751329s) [noise=true , address=145.79.227.9:4382 , piece_id=GFQZFT6GWRBLCNKCDIO26QNGW5TLPPUEFZLZ227UVW7O5KHV6CGQ, seg_pos={0 0}, upload=111828]: Done uploading piece. err: <nil> Dec 10 23:32:26.267957078 (128h3m26.139040133s) [noise=true , address=145.79.227.47:4268 , piece_id=K5BTBN3PS4K4WI77ER4BL73B6XB4SJOJNTSSFFCWM7FUJR2ESYNQ, seg_pos={0 0}, upload=111828]: Done uploading piece. err: <nil> Dec 10 23:32:26.270601253 (128h3m26.141684308s) [noise=true , address=145.79.227.41:4118 , piece_id=7SIIFPT7FDYBJGU7BKNJAH7QL5ACZTXIE6EAZ35X2Q473M36TRAA, seg_pos={0 0}, upload=111828]: Done uploading piece. err: <nil> Dec 10 23:32:26.291600608 (128h3m26.162683663s) [noise=true , address=145.79.227.49:4365 , piece_id=3CDJOGWEAZJ6OWKS2ZTOX33CN7VU76423B2WU5F4RWNIZDMYANSA, seg_pos={0 0}, upload=111828]: Done uploading piece. err: <nil> Dec 10 23:32:26.303446976 (128h3m26.174530011s) [noise=true , address=145.79.227.7:4296 , piece_id=ANDIF4JW2GAUYO6L45QBMEAOC36T77FOENXMM2C6DPNEPABU4BPQ, seg_pos={0 0}, upload=111828]: Done uploading piece. err: <nil> Dec 10 23:32:26.307108357 (128h3m26.178191392s) [noise=true , address=145.79.227.25:4628 , piece_id=6WZWP6O4Z6DVL4MV4OJVC5FEHJ3AKKBTDWUI3BZGKL674JNMXKTA, seg_pos={0 0}, upload=111828]: Done uploading piece. err: <nil> Dec 10 23:32:26.415229125 (128h3m26.28631217s) [noise=true , address=45.91.116.6:4990 , piece_id=SENJU357NGTDHZNWOCH2AUNNWKVV2DUYR3SDO5GNDDRZQ3XEHLXQ, seg_pos={0 0}, upload=111828]: Done uploading piece. err: <nil> Dec 10 23:32:26.416759359 (128h3m26.287842404s) [noise=true , address=45.91.116.5:4322 , piece_id=3CKTYVMA2XG7LJWNF6E26QFR3RQQFQKMNYAWVKOERQOLMMN5X43A, seg_pos={0 0}, upload=111828]: Done uploading piece. err: <nil> Dec 10 23:32:26.445015052 (128h3m26.316098097s) [noise=true , address=45.91.116.4:4963 , piece_id=DP4IKIY3BRDEGMCV3LHSUQHBF2WMGENSQ5OMYUZDX3Z65JAO564Q, seg_pos={0 0}, upload=111828]: Done uploading piece. err: <nil> Dec 10 23:32:26.446562719 (128h3m26.317645754s) [noise=true , address=45.91.116.12:4246 , piece_id=JPQNIW7L776ZZZSDJWI2AHMMHQ57LHG6MQXMXXKIWRC7M64TP3RA, seg_pos={0 0}, upload=111828]: Done uploading piece. err: <nil> Dec 10 23:33:23.458907169 (128h4m23.329990224s) [noise=true , address=77.90.31.16:4749 , piece_id=OP4KBGPYLUCOVLXPKMSP2A2TMR2WU5KZUKNBLGEEITVP5GCLGZNA, seg_pos={0 0}, upload=111828]: Done uploading piece. err: ecclient: upload failed (node:1EWSdFD1GDpqBHwXHZiDzR9nj8wE4vzmwnMbTrXrveiQXkUVC7, address:77.90.31.16:4749): protocol: rpc: dial tcp 77.90.31.16:4749: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:33:23.459259399 (128h4m23.330342434s) [noise=true , address=77.90.31.15:4728 , piece_id=5S3AZ2ASGL4W4O5M243GYHVH6SNJVXNCGWURL237OZOKYV4PIV4Q, seg_pos={0 0}, upload=111828]: Done uploading piece. err: ecclient: upload failed (node:152bVnKGB6XVkvkjrWDsivCMxEPXq3tesh1srTzXZBcuEohFDo, address:77.90.31.15:4728): protocol: rpc: dial tcp 77.90.31.15:4728: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:33:23.459457797 (128h4m23.330540843s) [noise=true , address=77.90.31.15:4715 , piece_id=GSNDMPPZ6YXF2A3CORCGSIAJXX5UB5OSHRZGWZTKLMYVNRDS4HWA, seg_pos={0 0}, upload=111828]: Done uploading piece. err: ecclient: upload failed (node:12WXsguvsox5t3gJTmYgXxSvS9Wha4DSSzhfadhoASU4see1FCs, address:77.90.31.15:4715): protocol: rpc: dial tcp 77.90.31.15:4715: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:33:23.459645204 (128h4m23.330728249s) [noise=true , address=77.90.31.18:4838 , piece_id=MR3RYVKHQSNQDWEO5KJCBBH6CQKUFZELT5IUKH3RRLCOGOUN77KA, seg_pos={0 0}, upload=111828]: Done uploading piece. err: ecclient: upload failed (node:1if7vf1cNTBwAhb6owi6tuGLYqSfa3feWhsvGSevwuwykhz2RY, address:77.90.31.18:4838): protocol: rpc: dial tcp 77.90.31.18:4838: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:33:23.459821199 (128h4m23.330904234s) [noise=true , address=77.90.31.13:4630 , piece_id=6VPE544CN6FNHTS7GYOBN6MO7CCR4T7EEPXL2LB3HMD6H5DIOTNQ, seg_pos={0 0}, upload=111828]: Done uploading piece. err: ecclient: upload failed (node:1gyQJLnzWmRESyd8m6G7aZWsQwVfqRQNARxRmd7GwwfYwhuzVn, address:77.90.31.13:4630): protocol: rpc: dial tcp 77.90.31.13:4630: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:33:23.460563863 (128h4m23.331646898s) [noise=true , address=77.90.31.17:4780 , piece_id=263Z64K3MMOS27TK5MO33MSS5PBBLZ6UJGV44LNGN4TDB4YQDYGQ, seg_pos={0 0}, upload=111828]: Done uploading piece. err: ecclient: upload failed (node:1XfvmVdBudtFnKwNxCxuGbQMS6HSSkVwPkYmffa5X6e6uYNcuF, address:77.90.31.17:4780): protocol: context deadline exceeded; context deadline exceeded Dec 10 23:33:23.460764255 (128h4m23.33184729s) [noise=true , address=77.90.31.11:4564 , piece_id=BYIG5GF6AOBHKE6WYOGOTGHKP5HDYFNDBVIYHU5MNYEMCZVMO4HQ, seg_pos={0 0}, upload=111828]: Done uploading piece. err: ecclient: upload failed (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9, address:77.90.31.11:4564): protocol: context deadline exceeded; context deadline exceeded Dec 10 23:33:23.461034049 (128h4m23.332117094s) [noise=true , address=77.90.31.19:4859 , piece_id=GDDT7KP34HDFEDMVHETUOAK7KZFY2Z3UPZX4B2WONBCDJE66TITA, seg_pos={0 0}, upload=111828]: Done uploading piece. err: ecclient: upload failed (node:12Ydy1zHhZCa78uBWuQpgnvbZohVVPaUuaYAqdLpEJQzk423S3h, address:77.90.31.19:4859): protocol: context deadline exceeded; context deadline exceeded Dec 10 23:33:23.461061962 (128h4m23.332145007s) [seg_pos={0 0}, upload=111828]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Dec 10 23:33:23.699852874 (128h4m23.570935919s) [upload=111828]: Done uploading segments... Dec 10 23:33:23.699889815 (128h4m23.57097285s) [upload=111828]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 6.433633825s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 10 out of 60 is failed (threshold=3) [.................................................XXXXX.XXXXX] duration: 1m0.242585086s trace: https://jaeger.dev.storj.io/trace/5b106df65b5519b FAIL (2025-12-10 23:33:23 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== pload=111828]: Begin upload segment... Dec 10 23:32:24.390786338 (128h3m24.261869383s) [seg_pos={0 0}, upload=111828]: Done begin upload segment. Dec 10 23:32:24.390809502 (128h3m24.261892537s) [noise=true , address=145.79.227.9:4382 , piece_id=GFQZFT6GWRBLCNKCDIO26QNGW5TLPPUEFZLZ227UVW7O5KHV6CGQ, seg_pos={0 0}, upload=111828]: Uploading piece... Dec 10 23:32:24.390930392 (128h3m24.262013427s) [noise=true , address=77.90.31.16:4749 , piece_id=OP4KBGPYLUCOVLXPKMSP2A2TMR2WU5KZUKNBLGEEITVP5GCLGZNA, seg_pos={0 0}, upload=111828]: Uploading piece... Dec 10 23:32:24.391622149 (128h3m24.262705185s) [noise=true , address=77.90.31.18:4838 , piece_id=MR3RYVKHQSNQDWEO5KJCBBH6CQKUFZELT5IUKH3RRLCOGOUN77KA, seg_pos={0 0}, upload=111828]: Uploading piece... Dec 10 23:32:24.391938291 (128h3m24.263021326s) [noise=true , address=77.90.31.15:4728 , piece_id=5S3AZ2ASGL4W4O5M243GYHVH6SNJVXNCGWURL237OZOKYV4PIV4Q, seg_pos={0 0}, upload=111828]: Uploading piece... Dec 10 23:32:24.392227722 (128h3m24.263310757s) [noise=true , address=145.79.227.49:4365 , piece_id=3CDJOGWEAZJ6OWKS2ZTOX33CN7VU76423B2WU5F4RWNIZDMYANSA, seg_pos={0 0}, upload=111828]: Uploading piece... Dec 10 23:32:24.392239205 (128h3m24.26332224s) [noise=true , address=45.91.116.6:4990 , piece_id=SENJU357NGTDHZNWOCH2AUNNWKVV2DUYR3SDO5GNDDRZQ3XEHLXQ, seg_pos={0 0}, upload=111828]: Uploading piece... Dec 10 23:32:24.392594541 (128h3m24.263677586s) [noise=true , address=77.90.31.15:4715 , piece_id=GSNDMPPZ6YXF2A3CORCGSIAJXX5UB5OSHRZGWZTKLMYVNRDS4HWA, seg_pos={0 0}, upload=111828]: Uploading piece... Dec 10 23:32:24.392661579 (128h3m24.263744624s) [noise=true , address=145.79.227.47:4268 , piece_id=K5BTBN3PS4K4WI77ER4BL73B6XB4SJOJNTSSFFCWM7FUJR2ESYNQ, seg_pos={0 0}, upload=111828]: Uploading piece... Dec 10 23:32:24.392952142 (128h3m24.264035197s) [noise=true , address=45.91.116.5:4322 , piece_id=3CKTYVMA2XG7LJWNF6E26QFR3RQQFQKMNYAWVKOERQOLMMN5X43A, seg_pos={0 0}, upload=111828]: Uploading piece... Dec 10 23:32:24.392981077 (128h3m24.264064112s) [noise=true , address=145.79.227.41:4118 , piece_id=7SIIFPT7FDYBJGU7BKNJAH7QL5ACZTXIE6EAZ35X2Q473M36TRAA, seg_pos={0 0}, upload=111828]: Uploading piece... Dec 10 23:32:24.393296888 (128h3m24.264379923s) [noise=true , address=145.79.227.25:4628 , piece_id=6WZWP6O4Z6DVL4MV4OJVC5FEHJ3AKKBTDWUI3BZGKL674JNMXKTA, seg_pos={0 0}, upload=111828]: Uploading piece... Dec 10 23:32:24.393314242 (128h3m24.264397277s) [noise=true , address=77.90.31.11:4564 , piece_id=BYIG5GF6AOBHKE6WYOGOTGHKP5HDYFNDBVIYHU5MNYEMCZVMO4HQ, seg_pos={0 0}, upload=111828]: Uploading piece... Dec 10 23:32:24.393584496 (128h3m24.264667541s) [noise=true , address=77.90.31.13:4630 , piece_id=6VPE544CN6FNHTS7GYOBN6MO7CCR4T7EEPXL2LB3HMD6H5DIOTNQ, seg_pos={0 0}, upload=111828]: Uploading piece... Dec 10 23:32:24.393648378 (128h3m24.264731423s) [noise=true , address=45.91.116.4:4963 , piece_id=DP4IKIY3BRDEGMCV3LHSUQHBF2WMGENSQ5OMYUZDX3Z65JAO564Q, seg_pos={0 0}, upload=111828]: Uploading piece... Dec 10 23:32:24.394019685 (128h3m24.26510272s) [noise=true , address=45.91.116.12:4246 , piece_id=JPQNIW7L776ZZZSDJWI2AHMMHQ57LHG6MQXMXXKIWRC7M64TP3RA, seg_pos={0 0}, upload=111828]: Uploading piece... Dec 10 23:32:24.394444433 (128h3m24.265527479s) [noise=true , address=77.90.31.19:4859 , piece_id=GDDT7KP34HDFEDMVHETUOAK7KZFY2Z3UPZX4B2WONBCDJE66TITA, seg_pos={0 0}, upload=111828]: Uploading piece... Dec 10 23:32:24.394933835 (128h3m24.26601689s) [noise=true , address=77.90.31.17:4780 , piece_id=263Z64K3MMOS27TK5MO33MSS5PBBLZ6UJGV44LNGN4TDB4YQDYGQ, seg_pos={0 0}, upload=111828]: Uploading piece... Dec 10 23:32:24.397086755 (128h3m24.26816979s) [noise=true , address=145.79.227.7:4296 , piece_id=ANDIF4JW2GAUYO6L45QBMEAOC36T77FOENXMM2C6DPNEPABU4BPQ, seg_pos={0 0}, upload=111828]: Uploading piece... Dec 10 23:32:25.861307488 (128h3m25.732390533s) [upload=111828]: Next returned nil segment Dec 10 23:32:25.861325311 (128h3m25.732408356s) [upload=111828]: Waiting for error group managing segments... Dec 10 23:32:26.249668283 (128h3m26.120751329s) [noise=true , address=145.79.227.9:4382 , piece_id=GFQZFT6GWRBLCNKCDIO26QNGW5TLPPUEFZLZ227UVW7O5KHV6CGQ, seg_pos={0 0}, upload=111828]: Done uploading piece. err: <nil> Dec 10 23:32:26.267957078 (128h3m26.139040133s) [noise=true , address=145.79.227.47:4268 , piece_id=K5BTBN3PS4K4WI77ER4BL73B6XB4SJOJNTSSFFCWM7FUJR2ESYNQ, seg_pos={0 0}, upload=111828]: Done uploading piece. err: <nil> Dec 10 23:32:26.270601253 (128h3m26.141684308s) [noise=true , address=145.79.227.41:4118 , piece_id=7SIIFPT7FDYBJGU7BKNJAH7QL5ACZTXIE6EAZ35X2Q473M36TRAA, seg_pos={0 0}, upload=111828]: Done uploading piece. err: <nil> Dec 10 23:32:26.291600608 (128h3m26.162683663s) [noise=true , address=145.79.227.49:4365 , piece_id=3CDJOGWEAZJ6OWKS2ZTOX33CN7VU76423B2WU5F4RWNIZDMYANSA, seg_pos={0 0}, upload=111828]: Done uploading piece. err: <nil> Dec 10 23:32:26.303446976 (128h3m26.174530011s) [noise=true , address=145.79.227.7:4296 , piece_id=ANDIF4JW2GAUYO6L45QBMEAOC36T77FOENXMM2C6DPNEPABU4BPQ, seg_pos={0 0}, upload=111828]: Done uploading piece. err: <nil> Dec 10 23:32:26.307108357 (128h3m26.178191392s) [noise=true , address=145.79.227.25:4628 , piece_id=6WZWP6O4Z6DVL4MV4OJVC5FEHJ3AKKBTDWUI3BZGKL674JNMXKTA, seg_pos={0 0}, upload=111828]: Done uploading piece. err: <nil> Dec 10 23:32:26.415229125 (128h3m26.28631217s) [noise=true , address=45.91.116.6:4990 , piece_id=SENJU357NGTDHZNWOCH2AUNNWKVV2DUYR3SDO5GNDDRZQ3XEHLXQ, seg_pos={0 0}, upload=111828]: Done uploading piece. err: <nil> Dec 10 23:32:26.416759359 (128h3m26.287842404s) [noise=true , address=45.91.116.5:4322 , piece_id=3CKTYVMA2XG7LJWNF6E26QFR3RQQFQKMNYAWVKOERQOLMMN5X43A, seg_pos={0 0}, upload=111828]: Done uploading piece. err: <nil> Dec 10 23:32:26.445015052 (128h3m26.316098097s) [noise=true , address=45.91.116.4:4963 , piece_id=DP4IKIY3BRDEGMCV3LHSUQHBF2WMGENSQ5OMYUZDX3Z65JAO564Q, seg_pos={0 0}, upload=111828]: Done uploading piece. err: <nil> Dec 10 23:32:26.446562719 (128h3m26.317645754s) [noise=true , address=45.91.116.12:4246 , piece_id=JPQNIW7L776ZZZSDJWI2AHMMHQ57LHG6MQXMXXKIWRC7M64TP3RA, seg_pos={0 0}, upload=111828]: Done uploading piece. err: <nil> Dec 10 23:33:23.458907169 (128h4m23.329990224s) [noise=true , address=77.90.31.16:4749 , piece_id=OP4KBGPYLUCOVLXPKMSP2A2TMR2WU5KZUKNBLGEEITVP5GCLGZNA, seg_pos={0 0}, upload=111828]: Done uploading piece. err: ecclient: upload failed (node:1EWSdFD1GDpqBHwXHZiDzR9nj8wE4vzmwnMbTrXrveiQXkUVC7, address:77.90.31.16:4749): protocol: rpc: dial tcp 77.90.31.16:4749: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:33:23.459259399 (128h4m23.330342434s) [noise=true , address=77.90.31.15:4728 , piece_id=5S3AZ2ASGL4W4O5M243GYHVH6SNJVXNCGWURL237OZOKYV4PIV4Q, seg_pos={0 0}, upload=111828]: Done uploading piece. err: ecclient: upload failed (node:152bVnKGB6XVkvkjrWDsivCMxEPXq3tesh1srTzXZBcuEohFDo, address:77.90.31.15:4728): protocol: rpc: dial tcp 77.90.31.15:4728: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:33:23.459457797 (128h4m23.330540843s) [noise=true , address=77.90.31.15:4715 , piece_id=GSNDMPPZ6YXF2A3CORCGSIAJXX5UB5OSHRZGWZTKLMYVNRDS4HWA, seg_pos={0 0}, upload=111828]: Done uploading piece. err: ecclient: upload failed (node:12WXsguvsox5t3gJTmYgXxSvS9Wha4DSSzhfadhoASU4see1FCs, address:77.90.31.15:4715): protocol: rpc: dial tcp 77.90.31.15:4715: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:33:23.459645204 (128h4m23.330728249s) [noise=true , address=77.90.31.18:4838 , piece_id=MR3RYVKHQSNQDWEO5KJCBBH6CQKUFZELT5IUKH3RRLCOGOUN77KA, seg_pos={0 0}, upload=111828]: Done uploading piece. err: ecclient: upload failed (node:1if7vf1cNTBwAhb6owi6tuGLYqSfa3feWhsvGSevwuwykhz2RY, address:77.90.31.18:4838): protocol: rpc: dial tcp 77.90.31.18:4838: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:33:23.459821199 (128h4m23.330904234s) [noise=true , address=77.90.31.13:4630 , piece_id=6VPE544CN6FNHTS7GYOBN6MO7CCR4T7EEPXL2LB3HMD6H5DIOTNQ, seg_pos={0 0}, upload=111828]: Done uploading piece. err: ecclient: upload failed (node:1gyQJLnzWmRESyd8m6G7aZWsQwVfqRQNARxRmd7GwwfYwhuzVn, address:77.90.31.13:4630): protocol: rpc: dial tcp 77.90.31.13:4630: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:33:23.460563863 (128h4m23.331646898s) [noise=true , address=77.90.31.17:4780 , piece_id=263Z64K3MMOS27TK5MO33MSS5PBBLZ6UJGV44LNGN4TDB4YQDYGQ, seg_pos={0 0}, upload=111828]: Done uploading piece. err: ecclient: upload failed (node:1XfvmVdBudtFnKwNxCxuGbQMS6HSSkVwPkYmffa5X6e6uYNcuF, address:77.90.31.17:4780): protocol: context deadline exceeded; context deadline exceeded Dec 10 23:33:23.460764255 (128h4m23.33184729s) [noise=true , address=77.90.31.11:4564 , piece_id=BYIG5GF6AOBHKE6WYOGOTGHKP5HDYFNDBVIYHU5MNYEMCZVMO4HQ, seg_pos={0 0}, upload=111828]: Done uploading piece. err: ecclient: upload failed (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9, address:77.90.31.11:4564): protocol: context deadline exceeded; context deadline exceeded Dec 10 23:33:23.461034049 (128h4m23.332117094s) [noise=true , address=77.90.31.19:4859 , piece_id=GDDT7KP34HDFEDMVHETUOAK7KZFY2Z3UPZX4B2WONBCDJE66TITA, seg_pos={0 0}, upload=111828]: Done uploading piece. err: ecclient: upload failed (node:12Ydy1zHhZCa78uBWuQpgnvbZohVVPaUuaYAqdLpEJQzk423S3h, address:77.90.31.19:4859): protocol: context deadline exceeded; context deadline exceeded Dec 10 23:33:23.461061962 (128h4m23.332145007s) [seg_pos={0 0}, upload=111828]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Dec 10 23:33:23.699852874 (128h4m23.570935919s) [upload=111828]: Done uploading segments... Dec 10 23:33:23.699889815 (128h4m23.57097285s) [upload=111828]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 1m0.242600606s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [....................................X.......................] duration: 4.512875174s trace: https://jaeger.dev.storj.io/trace/30cdf86753ce10d1 FAIL (2025-12-10 23:33:07 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 4.512944376s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 9 out of 60 is failed (threshold=3) [..................................................XXXXX.XXXX] duration: 1m0.240314626s trace: https://jaeger.dev.storj.io/trace/5276487b53873a99 FAIL (2025-12-10 23:32:23 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 489 (128h2m23.542279524s) [upload=111812]: Uploading segments... Dec 10 23:31:23.671339111 (128h2m23.542422156s) [upload=111812]: Got next segment. Inline: false Dec 10 23:31:23.671365521 (128h2m23.542448566s) [upload=111812]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 10 23:31:23.671372304 (128h2m23.542455339s) [upload=111812]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 10 23:31:23.910067305 (128h2m23.78115034s) [seg_pos={0 0}, upload=111812]: Begin upload segment... Dec 10 23:31:23.910122109 (128h2m23.781205154s) [seg_pos={0 0}, upload=111812]: Done begin upload segment. Dec 10 23:31:23.910169830 (128h2m23.781252865s) [noise=true , address=145.79.227.20:4472 , piece_id=N6BLOPGEY254DN4AWEIFVQWVGJUYZIJBYJY3L5Z7NWU7U63TFWZA, seg_pos={0 0}, upload=111812]: Uploading piece... Dec 10 23:31:23.910191121 (128h2m23.781274166s) [noise=true , address=77.90.31.16:4740 , piece_id=RQMB4XEUJZUS7J7QWW6ONAZIJZPQF7IG4PHTKAF6YRFJQIADAOLA, seg_pos={0 0}, upload=111812]: Uploading piece... Dec 10 23:31:23.910787206 (128h2m23.781870241s) [noise=true , address=64.49.13.169:4708 , piece_id=K66IF3IRTUK4HYOTWEQML5L6NSYPCCMTTZPXME363JKOH7335Z7Q, seg_pos={0 0}, upload=111812]: Uploading piece... Dec 10 23:31:23.911128065 (128h2m23.7822111s) [noise=true , address=45.91.116.12:4228 , piece_id=KWDFM3NXYWPUBFCOA3UV235ADPATVSIFA6DEBNWX6EMGFUHLUX2Q, seg_pos={0 0}, upload=111812]: Uploading piece... Dec 10 23:31:23.911497759 (128h2m23.782580804s) [noise=true , address=77.90.31.17:4804 , piece_id=JT5YYFTRK6W7VLXI3KKCEQIW62OQIX44GNBUX7FM47PMB73GEBIA, seg_pos={0 0}, upload=111812]: Uploading piece... Dec 10 23:31:23.911827296 (128h2m23.782910341s) [noise=true , address=45.91.116.9:4280 , piece_id=JKTZ5AZLNBUJ4FLIAFYHTZHH6I3KVUCDAYPQA64NEOFWMMS67FOQ, seg_pos={0 0}, upload=111812]: Uploading piece... Dec 10 23:31:23.912185278 (128h2m23.783268313s) [noise=true , address=77.90.31.13:4650 , piece_id=3TZMQ4ALXU54HHN4MJHNJCEEKU5ND6C67GH5VHBI3PIJSPNVARQQ, seg_pos={0 0}, upload=111812]: Uploading piece... Dec 10 23:31:23.912515948 (128h2m23.783598993s) [noise=true , address=64.49.13.26:4951 , piece_id=BDC2QNUVKZL57RDWPEGWF46CY4GPKKSCQSH5EJP7ENNM7MOELJJQ, seg_pos={0 0}, upload=111812]: Uploading piece... Dec 10 23:31:23.912888907 (128h2m23.783971942s) [noise=true , address=45.91.116.10:4096 , piece_id=YKTG6QZQ2MOUWKYPEVC3HB2TN4I4BCMJMQTQD2HE6UI5LE4MKAUA, seg_pos={0 0}, upload=111812]: Uploading piece... Dec 10 23:31:23.913274843 (128h2m23.784357878s) [noise=true , address=77.90.31.19:4867 , piece_id=WJJNFTC6DYDJMGLGIUEBFMHHIRXPVGD3SA62UOS4JUFUUAPXBJXA, seg_pos={0 0}, upload=111812]: Uploading piece... Dec 10 23:31:23.914318460 (128h2m23.785401495s) [noise=true , address=145.79.227.31:4802 , piece_id=GU7XAWANBK4QIH3ESPOPYJXFKCR5VKIYEQP5O6AWLNTKAPEWPAKQ, seg_pos={0 0}, upload=111812]: Uploading piece... Dec 10 23:31:23.915013464 (128h2m23.786096499s) [noise=true , address=45.91.116.11:4150 , piece_id=GAOHPLN5B3NAHJZ5U66H5K5R6GDHK7RQEJQ2OVDYPNG4J3DWBQ5Q, seg_pos={0 0}, upload=111812]: Uploading piece... Dec 10 23:31:23.915428714 (128h2m23.786511749s) [noise=true , address=145.79.227.15:4897 , piece_id=W7ZUSZTACX42M56Y3WSJUBIS2WMRXPVAXEP62V556JGIWORWJLBA, seg_pos={0 0}, upload=111812]: Uploading piece... Dec 10 23:31:23.915907235 (128h2m23.78699027s) [noise=true , address=77.90.31.21:4892 , piece_id=5LDL6VJCURSD7DJY5PXH7V5I7DJOBGFFXGVVAGW7XZKGQWJTU4XQ, seg_pos={0 0}, upload=111812]: Uploading piece... Dec 10 23:31:23.915963853 (128h2m23.787046898s) [noise=true , address=45.91.116.13:4178 , piece_id=EC7EDZAFKMAXUKKKYEYR64M43IRTTXCNI3GL75MLCFHQYXCAOXAQ, seg_pos={0 0}, upload=111812]: Uploading piece... Dec 10 23:31:23.916425282 (128h2m23.787508327s) [noise=true , address=145.79.227.30:4767 , piece_id=7IDDRAO65C3SMAFBN6RUD37Y3COHTQ5DJOPXYXN7CSJGFXU3FS6Q, seg_pos={0 0}, upload=111812]: Uploading piece... Dec 10 23:31:23.916801628 (128h2m23.787884674s) [noise=true , address=145.79.227.42:4131 , piece_id=YNCICW6I24C4I3HAUFABP2LRHQSBFXQO7LLVUMDW2CZLG76A237Q, seg_pos={0 0}, upload=111812]: Uploading piece... Dec 10 23:31:23.917195648 (128h2m23.788278683s) [noise=true , address=145.79.227.22:4532 , piece_id=NWIF6CLXT2N3SOR6WPYIKBJWAN2FX73TYPCVURNDQYXSWE6RJYTQ, seg_pos={0 0}, upload=111812]: Uploading piece... Dec 10 23:31:25.371004216 (128h2m25.242087261s) [upload=111812]: Next returned nil segment Dec 10 23:31:25.371024174 (128h2m25.242107219s) [upload=111812]: Waiting for error group managing segments... Dec 10 23:31:25.783209026 (128h2m25.654292061s) [noise=true , address=145.79.227.20:4472 , piece_id=N6BLOPGEY254DN4AWEIFVQWVGJUYZIJBYJY3L5Z7NWU7U63TFWZA, seg_pos={0 0}, upload=111812]: Done uploading piece. err: <nil> Dec 10 23:31:25.791670797 (128h2m25.662753832s) [noise=true , address=145.79.227.31:4802 , piece_id=GU7XAWANBK4QIH3ESPOPYJXFKCR5VKIYEQP5O6AWLNTKAPEWPAKQ, seg_pos={0 0}, upload=111812]: Done uploading piece. err: <nil> Dec 10 23:31:25.795172616 (128h2m25.666255661s) [noise=true , address=145.79.227.15:4897 , piece_id=W7ZUSZTACX42M56Y3WSJUBIS2WMRXPVAXEP62V556JGIWORWJLBA, seg_pos={0 0}, upload=111812]: Done uploading piece. err: <nil> Dec 10 23:31:25.797188244 (128h2m25.668271289s) [noise=true , address=64.49.13.26:4951 , piece_id=BDC2QNUVKZL57RDWPEGWF46CY4GPKKSCQSH5EJP7ENNM7MOELJJQ, seg_pos={0 0}, upload=111812]: Done uploading piece. err: <nil> Dec 10 23:31:25.819508355 (128h2m25.69059139s) [noise=true , address=145.79.227.42:4131 , piece_id=YNCICW6I24C4I3HAUFABP2LRHQSBFXQO7LLVUMDW2CZLG76A237Q, seg_pos={0 0}, upload=111812]: Done uploading piece. err: <nil> Dec 10 23:31:25.821031174 (128h2m25.692114209s) [noise=true , address=145.79.227.22:4532 , piece_id=NWIF6CLXT2N3SOR6WPYIKBJWAN2FX73TYPCVURNDQYXSWE6RJYTQ, seg_pos={0 0}, upload=111812]: Done uploading piece. err: <nil> Dec 10 23:31:25.911515500 (128h2m25.782598536s) [noise=true , address=45.91.116.12:4228 , piece_id=KWDFM3NXYWPUBFCOA3UV235ADPATVSIFA6DEBNWX6EMGFUHLUX2Q, seg_pos={0 0}, upload=111812]: Done uploading piece. err: <nil> Dec 10 23:31:25.932341075 (128h2m25.80342411s) [noise=true , address=45.91.116.11:4150 , piece_id=GAOHPLN5B3NAHJZ5U66H5K5R6GDHK7RQEJQ2OVDYPNG4J3DWBQ5Q, seg_pos={0 0}, upload=111812]: Done uploading piece. err: <nil> Dec 10 23:31:25.951750623 (128h2m25.822833678s) [noise=true , address=45.91.116.10:4096 , piece_id=YKTG6QZQ2MOUWKYPEVC3HB2TN4I4BCMJMQTQD2HE6UI5LE4MKAUA, seg_pos={0 0}, upload=111812]: Done uploading piece. err: <nil> Dec 10 23:31:25.967157060 (128h2m25.838240105s) [noise=true , address=45.91.116.9:4280 , piece_id=JKTZ5AZLNBUJ4FLIAFYHTZHH6I3KVUCDAYPQA64NEOFWMMS67FOQ, seg_pos={0 0}, upload=111812]: Done uploading piece. err: <nil> Dec 10 23:31:25.975111045 (128h2m25.846194091s) [noise=true , address=45.91.116.13:4178 , piece_id=EC7EDZAFKMAXUKKKYEYR64M43IRTTXCNI3GL75MLCFHQYXCAOXAQ, seg_pos={0 0}, upload=111812]: Done uploading piece. err: <nil> Dec 10 23:31:26.716677438 (128h2m26.587760483s) [noise=true , address=145.79.227.30:4767 , piece_id=7IDDRAO65C3SMAFBN6RUD37Y3COHTQ5DJOPXYXN7CSJGFXU3FS6Q, seg_pos={0 0}, upload=111812]: Done uploading piece. err: <nil> Dec 10 23:31:40.632051608 (128h2m40.503134643s) [noise=true , address=64.49.13.169:4708 , piece_id=K66IF3IRTUK4HYOTWEQML5L6NSYPCCMTTZPXME363JKOH7335Z7Q, seg_pos={0 0}, upload=111812]: Done uploading piece. err: <nil> Dec 10 23:32:22.995958346 (128h3m22.867041392s) [noise=true , address=77.90.31.19:4867 , piece_id=WJJNFTC6DYDJMGLGIUEBFMHHIRXPVGD3SA62UOS4JUFUUAPXBJXA, seg_pos={0 0}, upload=111812]: Done uploading piece. err: ecclient: upload failed (node:1gczGyJMCtqUfaac5oy7o4CTwypVF8AZjrAAjoJweU3kBFbuXw, address:77.90.31.19:4867): protocol: rpc: dial tcp 77.90.31.19:4867: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:32:22.996022369 (128h3m22.867105414s) [noise=true , address=77.90.31.21:4892 , piece_id=5LDL6VJCURSD7DJY5PXH7V5I7DJOBGFFXGVVAGW7XZKGQWJTU4XQ, seg_pos={0 0}, upload=111812]: Done uploading piece. err: ecclient: upload failed (node:1q5ZaXkQ2kzSdKoqNhFzyNuYbWZ2uQern3umD3HtDYgTCSkGAa, address:77.90.31.21:4892): protocol: rpc: dial tcp 77.90.31.21:4892: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:32:22.996243470 (128h3m22.867326515s) [noise=true , address=77.90.31.16:4740 , piece_id=RQMB4XEUJZUS7J7QWW6ONAZIJZPQF7IG4PHTKAF6YRFJQIADAOLA, seg_pos={0 0}, upload=111812]: Done uploading piece. err: ecclient: upload failed (node:145raYQJozNw3SMbj78wDsR6EjPvJTqqcocf1SaPs3HEnNVS5p, address:77.90.31.16:4740): protocol: rpc: dial tcp 77.90.31.16:4740: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:32:22.996260382 (128h3m22.867343427s) [noise=true , address=77.90.31.17:4804 , piece_id=JT5YYFTRK6W7VLXI3KKCEQIW62OQIX44GNBUX7FM47PMB73GEBIA, seg_pos={0 0}, upload=111812]: Done uploading piece. err: ecclient: upload failed (node:168Tw7W69Tq9opDKhJ4cM8PGnzpxp9E9SxKbdVxJeF83pETqGx, address:77.90.31.17:4804): protocol: rpc: dial tcp 77.90.31.17:4804: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:32:22.996460493 (128h3m22.867543528s) [noise=true , address=77.90.31.13:4650 , piece_id=3TZMQ4ALXU54HHN4MJHNJCEEKU5ND6C67GH5VHBI3PIJSPNVARQQ, seg_pos={0 0}, upload=111812]: Done uploading piece. err: ecclient: upload failed (node:1FNXVNAv3KeiRqTnsvQkMobyRcNFmPAAjVqSNvgs6MPJH4smVU, address:77.90.31.13:4650): protocol: write tcp 188.166.225.170:43204->77.90.31.13:4650: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:32:22.996493075 (128h3m22.86757611s) [seg_pos={0 0}, upload=111812]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Dec 10 23:32:23.234842638 (128h3m23.105925693s) [upload=111812]: Done uploading segments... Dec 10 23:32:23.234883957 (128h3m23.105967002s) [upload=111812]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1m0.240326338s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.....................................X......................] duration: 4.030631853s trace: https://jaeger.dev.storj.io/trace/30cdf86753ce10d1 FAIL (2025-12-10 23:31:59 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 4.030679525s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [X.....................................X.....................] duration: 4.138619866s trace: https://jaeger.dev.storj.io/trace/30cdf86753ce10d1 FAIL (2025-12-10 23:31:43 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 4.13864309s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 8 out of 60 is failed (threshold=3) [...................................................XXXXX.XXX] duration: 1m0.241700931s trace: https://jaeger.dev.storj.io/trace/28d32191a6d8ba35 FAIL (2025-12-10 23:31:22 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== (128h1m23.090418576s) [upload=111798]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 10 23:30:23.468262886 (128h1m23.339345931s) [seg_pos={0 0}, upload=111798]: Begin upload segment... Dec 10 23:30:23.468323200 (128h1m23.339406245s) [seg_pos={0 0}, upload=111798]: Done begin upload segment. Dec 10 23:30:23.468360311 (128h1m23.339443356s) [noise=true , address=64.49.13.169:4708 , piece_id=PLRNMNYHEM7MYEYP55OGYM27KCZWD5XNMWRCIQSFMHFVSOGDVMRA, seg_pos={0 0}, upload=111798]: Uploading piece... Dec 10 23:30:23.470203811 (128h1m23.341286856s) [noise=true , address=5.231.73.4:4030 , piece_id=EBLIKWVDTATYXZWX3D76NSQAWAS4MZB4NB3CAYE4EUP2BQBREPTA, seg_pos={0 0}, upload=111798]: Uploading piece... Dec 10 23:30:23.470722528 (128h1m23.341805574s) [noise=true , address=45.91.116.6:4006 , piece_id=LCKPSO7LHE3BNBG3GHRYUNQIU7XOPYVGZQ6OBTZTP7FTTYGU5FAQ, seg_pos={0 0}, upload=111798]: Uploading piece... Dec 10 23:30:23.471130025 (128h1m23.34221306s) [noise=true , address=45.91.116.11:4127 , piece_id=572CEU53INSVWFYVYWLFJXGJNXU6SXHQ34XXWFDLFVQBXBJ7BA5Q, seg_pos={0 0}, upload=111798]: Uploading piece... Dec 10 23:30:23.471322340 (128h1m23.342405385s) [noise=true , address=145.79.227.32:4857 , piece_id=2E25TTNWCSKINRATFHFZ7NY25CG27WBVU426IXEAJWCQ3V2PD7CQ, seg_pos={0 0}, upload=111798]: Uploading piece... Dec 10 23:30:23.471426799 (128h1m23.342509844s) [noise=true , address=145.79.227.26:4657 , piece_id=AK6XCDS73NUEXF6NITPID2EW3S7YZFKKOWAMZLIL5XNESTFMFJUQ, seg_pos={0 0}, upload=111798]: Uploading piece... Dec 10 23:30:23.471723725 (128h1m23.34280676s) [noise=true , address=77.90.31.12:4623 , piece_id=FKQL6M2YBYHAROTIA6DCO2T6CQN2GINYNIYQAZAYXWPVTTUFWZLA, seg_pos={0 0}, upload=111798]: Uploading piece... Dec 10 23:30:23.471900682 (128h1m23.342983717s) [noise=true , address=145.79.227.50:4403 , piece_id=6OYGSHYUHIBDKGPGXQONHZNYKNCWKSYM7II6SKIEGHFCRQYPTJ5A, seg_pos={0 0}, upload=111798]: Uploading piece... Dec 10 23:30:23.472134227 (128h1m23.343217272s) [noise=true , address=45.91.116.10:4108 , piece_id=4UB5Z26ZDT2JQUF5IT5SJVRZMPUDPWXUA5SK5QV33XSC5UP7V3GA, seg_pos={0 0}, upload=111798]: Uploading piece... Dec 10 23:30:23.472475887 (128h1m23.343558932s) [noise=true , address=77.90.31.10:4537 , piece_id=M374NOYGLDT24UBOOPHEQQSE5G76MTRYYVZIYU5WBIVLT2U3M5WA, seg_pos={0 0}, upload=111798]: Uploading piece... Dec 10 23:30:23.472527996 (128h1m23.343611042s) [noise=true , address=45.91.116.5:4312 , piece_id=K6YQRYZJHEY5PFUTZFKRFWZSNVG7WBXHUMYPHJEG26L55IW4LZOA, seg_pos={0 0}, upload=111798]: Uploading piece... Dec 10 23:30:23.472947886 (128h1m23.344030931s) [noise=true , address=45.91.116.13:4184 , piece_id=RSEOCLJSTMHWI4VJTBSPIGOM5HOCURA4EYAXZ32U3FT3FZWCJJOA, seg_pos={0 0}, upload=111798]: Uploading piece... Dec 10 23:30:23.472983935 (128h1m23.34406698s) [noise=true , address=145.79.227.16:4932 , piece_id=6O3MECE75YJXW5QEQXB7G63X7J7CAAOLXQGFMO4NNZQGZBJNSFTQ, seg_pos={0 0}, upload=111798]: Uploading piece... Dec 10 23:30:23.473431908 (128h1m23.344514953s) [noise=true , address=77.90.31.15:4725 , piece_id=VHDVG6HAL4EVQZQ226L2JWXE4T2L5SLBIVMYYUTVNOSYIM3N7ZVA, seg_pos={0 0}, upload=111798]: Uploading piece... Dec 10 23:30:23.473565983 (128h1m23.344649018s) [noise=true , address=77.90.31.11:4568 , piece_id=66YXMP3Q6DVQBKLQ4XA5G4ILRTW4JMNZ4QZSLJKLDBOAEWRJBRXQ, seg_pos={0 0}, upload=111798]: Uploading piece... Dec 10 23:30:23.473905028 (128h1m23.344988063s) [noise=true , address=45.91.116.4:4954 , piece_id=QPUH4IUS43O5YERZPSYOUJFR4PZDJWAON5PVFYDLJWM2NXE5PFYQ, seg_pos={0 0}, upload=111798]: Uploading piece... Dec 10 23:30:23.474153100 (128h1m23.345236145s) [noise=true , address=77.90.31.17:4796 , piece_id=2RLIE4PE2O7X6V5AI7KFEUFJITZLJNJCDDKDSIODPEU3HSRN7OLA, seg_pos={0 0}, upload=111798]: Uploading piece... Dec 10 23:30:23.476534155 (128h1m23.34761719s) [noise=true , address=77.90.31.16:4741 , piece_id=OVRWP6ALWSCNFAVA55BQDBQMXFMYT7JUHMAE5LVBWVTKFUCCS2EQ, seg_pos={0 0}, upload=111798]: Uploading piece... Dec 10 23:30:24.921157811 (128h1m24.792240846s) [upload=111798]: Next returned nil segment Dec 10 23:30:24.921176207 (128h1m24.792259252s) [upload=111798]: Waiting for error group managing segments... Dec 10 23:30:25.335373550 (128h1m25.206456585s) [noise=true , address=145.79.227.32:4857 , piece_id=2E25TTNWCSKINRATFHFZ7NY25CG27WBVU426IXEAJWCQ3V2PD7CQ, seg_pos={0 0}, upload=111798]: Done uploading piece. err: <nil> Dec 10 23:30:25.344467085 (128h1m25.21555012s) [noise=true , address=145.79.227.50:4403 , piece_id=6OYGSHYUHIBDKGPGXQONHZNYKNCWKSYM7II6SKIEGHFCRQYPTJ5A, seg_pos={0 0}, upload=111798]: Done uploading piece. err: <nil> Dec 10 23:30:25.345215090 (128h1m25.216298135s) [noise=true , address=145.79.227.26:4657 , piece_id=AK6XCDS73NUEXF6NITPID2EW3S7YZFKKOWAMZLIL5XNESTFMFJUQ, seg_pos={0 0}, upload=111798]: Done uploading piece. err: <nil> Dec 10 23:30:25.352725931 (128h1m25.223808986s) [noise=true , address=5.231.73.4:4030 , piece_id=EBLIKWVDTATYXZWX3D76NSQAWAS4MZB4NB3CAYE4EUP2BQBREPTA, seg_pos={0 0}, upload=111798]: Done uploading piece. err: <nil> Dec 10 23:30:25.440364729 (128h1m25.311447774s) [noise=true , address=45.91.116.6:4006 , piece_id=LCKPSO7LHE3BNBG3GHRYUNQIU7XOPYVGZQ6OBTZTP7FTTYGU5FAQ, seg_pos={0 0}, upload=111798]: Done uploading piece. err: <nil> Dec 10 23:30:25.467049300 (128h1m25.338132345s) [noise=true , address=45.91.116.4:4954 , piece_id=QPUH4IUS43O5YERZPSYOUJFR4PZDJWAON5PVFYDLJWM2NXE5PFYQ, seg_pos={0 0}, upload=111798]: Done uploading piece. err: <nil> Dec 10 23:30:25.477019336 (128h1m25.348102371s) [noise=true , address=45.91.116.10:4108 , piece_id=4UB5Z26ZDT2JQUF5IT5SJVRZMPUDPWXUA5SK5QV33XSC5UP7V3GA, seg_pos={0 0}, upload=111798]: Done uploading piece. err: <nil> Dec 10 23:30:25.498631588 (128h1m25.369714623s) [noise=true , address=45.91.116.5:4312 , piece_id=K6YQRYZJHEY5PFUTZFKRFWZSNVG7WBXHUMYPHJEG26L55IW4LZOA, seg_pos={0 0}, upload=111798]: Done uploading piece. err: <nil> Dec 10 23:30:25.503721109 (128h1m25.374804154s) [noise=true , address=45.91.116.11:4127 , piece_id=572CEU53INSVWFYVYWLFJXGJNXU6SXHQ34XXWFDLFVQBXBJ7BA5Q, seg_pos={0 0}, upload=111798]: Done uploading piece. err: <nil> Dec 10 23:30:25.552697748 (128h1m25.423780793s) [noise=true , address=45.91.116.13:4184 , piece_id=RSEOCLJSTMHWI4VJTBSPIGOM5HOCURA4EYAXZ32U3FT3FZWCJJOA, seg_pos={0 0}, upload=111798]: Done uploading piece. err: <nil> Dec 10 23:30:26.421586089 (128h1m26.292669135s) [noise=true , address=145.79.227.16:4932 , piece_id=6O3MECE75YJXW5QEQXB7G63X7J7CAAOLXQGFMO4NNZQGZBJNSFTQ, seg_pos={0 0}, upload=111798]: Done uploading piece. err: <nil> Dec 10 23:30:40.231573354 (128h1m40.102656399s) [noise=true , address=64.49.13.169:4708 , piece_id=PLRNMNYHEM7MYEYP55OGYM27KCZWD5XNMWRCIQSFMHFVSOGDVMRA, seg_pos={0 0}, upload=111798]: Done uploading piece. err: <nil> Dec 10 23:31:22.530097868 (128h2m22.401180953s) [noise=true , address=77.90.31.15:4725 , piece_id=VHDVG6HAL4EVQZQ226L2JWXE4T2L5SLBIVMYYUTVNOSYIM3N7ZVA, seg_pos={0 0}, upload=111798]: Done uploading piece. err: ecclient: upload failed (node:1nstPr2GMJo6pnBmBuDME5QYw3mXgnqLoaTKkcKUhjSbAY8mQk, address:77.90.31.15:4725): protocol: write tcp 188.166.225.170:58008->77.90.31.15:4725: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:31:22.530447453 (128h2m22.401530498s) [noise=true , address=77.90.31.12:4623 , piece_id=FKQL6M2YBYHAROTIA6DCO2T6CQN2GINYNIYQAZAYXWPVTTUFWZLA, seg_pos={0 0}, upload=111798]: Done uploading piece. err: ecclient: upload failed (node:1ANjFmZrb9SDcqEHW8CBzeMtwvmceoLH9MtPabAjBw8wXjBuaM, address:77.90.31.12:4623): protocol: write tcp 188.166.225.170:53686->77.90.31.12:4623: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:31:22.530552403 (128h2m22.401635468s) [noise=true , address=77.90.31.17:4796 , piece_id=2RLIE4PE2O7X6V5AI7KFEUFJITZLJNJCDDKDSIODPEU3HSRN7OLA, seg_pos={0 0}, upload=111798]: Done uploading piece. err: ecclient: upload failed (node:1KK6y5NBuZovne8n3gMvAyDcZFasZxDzorvPeHe7CefoFs27UF, address:77.90.31.17:4796): protocol: rpc: dial tcp 77.90.31.17:4796: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:31:22.530633728 (128h2m22.401716763s) [noise=true , address=77.90.31.10:4537 , piece_id=M374NOYGLDT24UBOOPHEQQSE5G76MTRYYVZIYU5WBIVLT2U3M5WA, seg_pos={0 0}, upload=111798]: Done uploading piece. err: ecclient: upload failed (node:12M5ger8kngGvMCD9c2qBxpFNCpLxka2fh8iWrf2XerxRiQEUja, address:77.90.31.10:4537): protocol: rpc: dial tcp 77.90.31.10:4537: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:31:22.530737425 (128h2m22.40182046s) [noise=true , address=77.90.31.11:4568 , piece_id=66YXMP3Q6DVQBKLQ4XA5G4ILRTW4JMNZ4QZSLJKLDBOAEWRJBRXQ, seg_pos={0 0}, upload=111798]: Done uploading piece. err: ecclient: upload failed (node:1H1L8DTKD7W7Xx1PVT7KzB9GkGBadnTEj9BHtbSgcW1S7Jj8Tw, address:77.90.31.11:4568): protocol: write tcp 188.166.225.170:56490->77.90.31.11:4568: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:31:22.530812649 (128h2m22.401895694s) [noise=true , address=77.90.31.16:4741 , piece_id=OVRWP6ALWSCNFAVA55BQDBQMXFMYT7JUHMAE5LVBWVTKFUCCS2EQ, seg_pos={0 0}, upload=111798]: Done uploading piece. err: ecclient: upload failed (node:12G8MCviRyA85FwyuN3sXF9pwWQnP6mki6Ux68Eefx42BeWUcNR, address:77.90.31.16:4741): protocol: rpc: dial tcp 77.90.31.16:4741: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:31:22.530842014 (128h2m22.401925059s) [seg_pos={0 0}, upload=111798]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Dec 10 23:31:22.768876468 (128h2m22.639959503s) [upload=111798]: Done uploading segments... Dec 10 23:31:22.768922757 (128h2m22.640005792s) [upload=111798]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration:* 1m0.241714388s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.X.....................................X....................] duration: 3.935350111s trace: https://jaeger.dev.storj.io/trace/30cdf86753ce10d1 FAIL (2025-12-10 23:31:13 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 3.935369607s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [..X.....................................X...................] duration: 4.163575674s trace: https://jaeger.dev.storj.io/trace/30cdf86753ce10d1 FAIL (2025-12-10 23:30:56 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 4.163598098s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [...X.....................................X..................] duration: 4.077762395s trace: https://jaeger.dev.storj.io/trace/30cdf86753ce10d1 FAIL (2025-12-10 23:30:38 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 4.077834654s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 7 out of 60 is failed (threshold=3) [....................................................XXXXX.XX] duration: 1m0.246072875s trace: https://jaeger.dev.storj.io/trace/49f789595dfcc098 FAIL (2025-12-10 23:30:22 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== : Got next segment. Inline: false Dec 10 23:29:22.749489428 (128h0m22.620572463s) [upload=111783]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 10 23:29:22.749494187 (128h0m22.620577232s) [upload=111783]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 10 23:29:22.994437294 (128h0m22.865520329s) [seg_pos={0 0}, upload=111783]: Begin upload segment... Dec 10 23:29:22.994493332 (128h0m22.865576367s) [seg_pos={0 0}, upload=111783]: Done begin upload segment. Dec 10 23:29:22.994547254 (128h0m22.865630299s) [noise=true , address=145.79.227.35:4502 , piece_id=US4LROIKJB5XC23Q2K5WMA5QH4SGRHSTG7PAJRDDG2FUHXW7WHXQ, seg_pos={0 0}, upload=111783]: Uploading piece... Dec 10 23:29:22.994632997 (128h0m22.865716053s) [noise=true , address=145.79.227.16:4938 , piece_id=OXO6AN2YLCQ25ED3LEF24JBGCJVPT52ODVRWYCPTUICZ3XIZ3JCA, seg_pos={0 0}, upload=111783]: Uploading piece... Dec 10 23:29:22.995333070 (128h0m22.866416105s) [noise=true , address=64.49.13.26:4954 , piece_id=H5CW5M2K5XPFCCXWOIKQ4V277PCFHSI4MJ7DOJGPP3PNRY6OMAVQ, seg_pos={0 0}, upload=111783]: Uploading piece... Dec 10 23:29:22.996073861 (128h0m22.867156896s) [noise=true , address=77.90.31.19:4861 , piece_id=M53GVJL6TJZMFY5MNHVKQNTZ3TE2PR4YQ37BXUGIB7KNDQXQ36IQ, seg_pos={0 0}, upload=111783]: Uploading piece... Dec 10 23:29:22.996930362 (128h0m22.868013407s) [noise=true , address=145.79.227.51:4424 , piece_id=CD7SXVQDMEQ46U2DLMAJWODKK2NYJMSQVZQ3754X3WLQCD5TG3VA, seg_pos={0 0}, upload=111783]: Uploading piece... Dec 10 23:29:22.997019972 (128h0m22.868103017s) [noise=true , address=77.90.31.16:4750 , piece_id=FXCHTNZ3UZZ2S7ZF2HWYOGTIEMYBRUU3REPYTB2KSPGCBRN3GO6Q, seg_pos={0 0}, upload=111783]: Uploading piece... Dec 10 23:29:22.997877445 (128h0m22.86896049s) [noise=true , address=45.91.116.4:4961 , piece_id=HONEUMIB7BUE75PSCYLXY6HF3JBG6LSON7LPSYRNDTO3TSNNIX6Q, seg_pos={0 0}, upload=111783]: Uploading piece... Dec 10 23:29:22.997934414 (128h0m22.869017449s) [noise=true , address=45.91.116.13:4187 , piece_id=AULTIT7Y4NX7PEPFAE7FPWTL4TA2CPFLW7EVM6TNZ22BLBWWXCCQ, seg_pos={0 0}, upload=111783]: Uploading piece... Dec 10 23:29:22.998692216 (128h0m22.869775251s) [noise=true , address=145.79.227.31:4800 , piece_id=ZXWY3JGMPCRDED4236XK3B73KLC2WUAR3DEKOWMOV4F6GVWCKBHA, seg_pos={0 0}, upload=111783]: Uploading piece... Dec 10 23:29:23.000129654 (128h0m22.871212709s) [noise=true , address=77.90.31.17:4798 , piece_id=VRZBS5FCJWAH2XYSD6BWBQ2ZFK2BFDPIMHTI4C7MLW5HVGUUDRUQ, seg_pos={0 0}, upload=111783]: Uploading piece... Dec 10 23:29:23.001125089 (128h0m22.872208134s) [noise=true , address=145.79.227.29:4967 , piece_id=L26SZBG3LZKLFPH3EMG5VLDSAVARVGBRWMJMROXBOOOVY5W7FNVQ, seg_pos={0 0}, upload=111783]: Uploading piece... Dec 10 23:29:23.001498841 (128h0m22.872581886s) [noise=true , address=45.91.116.6:4985 , piece_id=TACJFMG62T3YIEWTMSE3P5UC4G3E5LRVUKCESDO5GQFLQQVNNMHA, seg_pos={0 0}, upload=111783]: Uploading piece... Dec 10 23:29:23.001910866 (128h0m22.872993931s) [noise=true , address=45.91.116.14:4203 , piece_id=ZWRQYLRNU4RJME35HMIQN5UMD72KO2IGGM6ICQ3DQK35BPIBSUPQ, seg_pos={0 0}, upload=111783]: Uploading piece... Dec 10 23:29:23.002766655 (128h0m22.8738497s) [noise=true , address=77.90.31.21:4909 , piece_id=3WJKP6NWWMS5SDNVL54VTWNRJHVD7TWOR3YIWIRUFH3T7I5J4GFA, seg_pos={0 0}, upload=111783]: Uploading piece... Dec 10 23:29:23.003553083 (128h0m22.874636118s) [noise=true , address=145.79.227.48:4313 , piece_id=XML4DW7GDEX3E3CSMWSMVTHFTSGI7OFKU676C34XYJWQ53UOEBNA, seg_pos={0 0}, upload=111783]: Uploading piece... Dec 10 23:29:23.004277542 (128h0m22.875360587s) [noise=true , address=45.91.116.7:4039 , piece_id=6MVVMJSMYMCHILKEC5WXMJ2K7IMLTPBTHBDNJM3QHMUSTKMA6RSA, seg_pos={0 0}, upload=111783]: Uploading piece... Dec 10 23:29:23.004589536 (128h0m22.875672581s) [noise=true , address=77.90.31.11:4577 , piece_id=PXNWOLBLTT5L2FJECPTVFBHUFAL2BZ3R43L4XM75FMDSZ2UJBVVA, seg_pos={0 0}, upload=111783]: Uploading piece... Dec 10 23:29:23.005046867 (128h0m22.876129902s) [noise=true , address=145.79.227.16:4914 , piece_id=7R6Q63E6T4O3355F2VLEOEBTU5SVTHWSGDPLNSLLUANRULWMN3WA, seg_pos={0 0}, upload=111783]: Uploading piece... Dec 10 23:29:24.433980459 (128h0m24.305063504s) [upload=111783]: Next returned nil segment Dec 10 23:29:24.433994656 (128h0m24.305077701s) [upload=111783]: Waiting for error group managing segments... Dec 10 23:29:24.852418387 (128h0m24.723501423s) [noise=true , address=145.79.227.48:4313 , piece_id=XML4DW7GDEX3E3CSMWSMVTHFTSGI7OFKU676C34XYJWQ53UOEBNA, seg_pos={0 0}, upload=111783]: Done uploading piece. err: <nil> Dec 10 23:29:24.863238660 (128h0m24.734321705s) [noise=true , address=145.79.227.29:4967 , piece_id=L26SZBG3LZKLFPH3EMG5VLDSAVARVGBRWMJMROXBOOOVY5W7FNVQ, seg_pos={0 0}, upload=111783]: Done uploading piece. err: <nil> Dec 10 23:29:24.863358599 (128h0m24.734441634s) [noise=true , address=145.79.227.51:4424 , piece_id=CD7SXVQDMEQ46U2DLMAJWODKK2NYJMSQVZQ3754X3WLQCD5TG3VA, seg_pos={0 0}, upload=111783]: Done uploading piece. err: <nil> Dec 10 23:29:24.921538933 (128h0m24.792621968s) [noise=true , address=145.79.227.35:4502 , piece_id=US4LROIKJB5XC23Q2K5WMA5QH4SGRHSTG7PAJRDDG2FUHXW7WHXQ, seg_pos={0 0}, upload=111783]: Done uploading piece. err: <nil> Dec 10 23:29:24.923755373 (128h0m24.794838418s) [noise=true , address=64.49.13.26:4954 , piece_id=H5CW5M2K5XPFCCXWOIKQ4V277PCFHSI4MJ7DOJGPP3PNRY6OMAVQ, seg_pos={0 0}, upload=111783]: Done uploading piece. err: <nil> Dec 10 23:29:24.940027748 (128h0m24.811110783s) [noise=true , address=145.79.227.31:4800 , piece_id=ZXWY3JGMPCRDED4236XK3B73KLC2WUAR3DEKOWMOV4F6GVWCKBHA, seg_pos={0 0}, upload=111783]: Done uploading piece. err: <nil> Dec 10 23:29:24.986383296 (128h0m24.857466341s) [noise=true , address=45.91.116.4:4961 , piece_id=HONEUMIB7BUE75PSCYLXY6HF3JBG6LSON7LPSYRNDTO3TSNNIX6Q, seg_pos={0 0}, upload=111783]: Done uploading piece. err: <nil> Dec 10 23:29:25.030187266 (128h0m24.901270301s) [noise=true , address=45.91.116.6:4985 , piece_id=TACJFMG62T3YIEWTMSE3P5UC4G3E5LRVUKCESDO5GQFLQQVNNMHA, seg_pos={0 0}, upload=111783]: Done uploading piece. err: <nil> Dec 10 23:29:25.034198554 (128h0m24.905281589s) [noise=true , address=45.91.116.7:4039 , piece_id=6MVVMJSMYMCHILKEC5WXMJ2K7IMLTPBTHBDNJM3QHMUSTKMA6RSA, seg_pos={0 0}, upload=111783]: Done uploading piece. err: <nil> Dec 10 23:29:25.084645963 (128h0m24.955728998s) [noise=true , address=45.91.116.14:4203 , piece_id=ZWRQYLRNU4RJME35HMIQN5UMD72KO2IGGM6ICQ3DQK35BPIBSUPQ, seg_pos={0 0}, upload=111783]: Done uploading piece. err: <nil> Dec 10 23:29:25.156497560 (128h0m25.027580595s) [noise=true , address=45.91.116.13:4187 , piece_id=AULTIT7Y4NX7PEPFAE7FPWTL4TA2CPFLW7EVM6TNZ22BLBWWXCCQ, seg_pos={0 0}, upload=111783]: Done uploading piece. err: <nil> Dec 10 23:29:25.877309283 (128h0m25.748392328s) [noise=true , address=145.79.227.16:4914 , piece_id=7R6Q63E6T4O3355F2VLEOEBTU5SVTHWSGDPLNSLLUANRULWMN3WA, seg_pos={0 0}, upload=111783]: Done uploading piece. err: <nil> Dec 10 23:29:25.909202412 (128h0m25.780285457s) [noise=true , address=145.79.227.16:4938 , piece_id=OXO6AN2YLCQ25ED3LEF24JBGCJVPT52ODVRWYCPTUICZ3XIZ3JCA, seg_pos={0 0}, upload=111783]: Done uploading piece. err: <nil> Dec 10 23:30:22.066469885 (128h1m21.93755293s) [noise=true , address=77.90.31.21:4909 , piece_id=3WJKP6NWWMS5SDNVL54VTWNRJHVD7TWOR3YIWIRUFH3T7I5J4GFA, seg_pos={0 0}, upload=111783]: Done uploading piece. err: ecclient: upload failed (node:121RT7tP6czMFAiZ3DjzSSsYgPt3xZXVbXVpkcYxaZYscNiMjL6, address:77.90.31.21:4909): protocol: write tcp 188.166.225.170:39942->77.90.31.21:4909: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:30:22.066693510 (128h1m21.937776555s) [noise=true , address=77.90.31.17:4798 , piece_id=VRZBS5FCJWAH2XYSD6BWBQ2ZFK2BFDPIMHTI4C7MLW5HVGUUDRUQ, seg_pos={0 0}, upload=111783]: Done uploading piece. err: ecclient: upload failed (node:123HQXrHXyyMYHd5j7nqjzdGWoNg6m8j1eejUcvHNRfCRaHjAWK, address:77.90.31.17:4798): protocol: write tcp 188.166.225.170:55956->77.90.31.17:4798: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:30:22.066721914 (128h1m21.937804959s) [noise=true , address=77.90.31.11:4577 , piece_id=PXNWOLBLTT5L2FJECPTVFBHUFAL2BZ3R43L4XM75FMDSZ2UJBVVA, seg_pos={0 0}, upload=111783]: Done uploading piece. err: ecclient: upload failed (node:12rKdimacehmW1Tp5Zp4DJFxafLhcEeNh61cfw8Bfefr9pRG2ei, address:77.90.31.11:4577): protocol: write tcp 188.166.225.170:36938->77.90.31.11:4577: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:30:22.067083933 (128h1m21.938167008s) [noise=true , address=77.90.31.19:4861 , piece_id=M53GVJL6TJZMFY5MNHVKQNTZ3TE2PR4YQ37BXUGIB7KNDQXQ36IQ, seg_pos={0 0}, upload=111783]: Done uploading piece. err: ecclient: upload failed (node:12MXMA4zbRGtLCJaAfF2BchSfazrAR7ToZnpazHkNEapzzm44QQ, address:77.90.31.19:4861): protocol: rpc: dial tcp 77.90.31.19:4861: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:30:22.067304474 (128h1m21.938387509s) [noise=true , address=77.90.31.16:4750 , piece_id=FXCHTNZ3UZZ2S7ZF2HWYOGTIEMYBRUU3REPYTB2KSPGCBRN3GO6Q, seg_pos={0 0}, upload=111783]: Done uploading piece. err: ecclient: upload failed (node:12TGd8NnWdekDtHzBQHmJPgTtHjXC6g8fi6RoZYFQJVHDa5hdyE, address:77.90.31.16:4750): protocol: write tcp 188.166.225.170:53894->77.90.31.16:4750: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:30:22.067343438 (128h1m21.938426483s) [seg_pos={0 0}, upload=111783]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Dec 10 23:30:22.309106701 (128h1m22.180189756s) [upload=111783]: Done uploading segments... Dec 10 23:30:22.309155994 (128h1m22.180239039s) [upload=111783]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1m0.246109926s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [....X.....................................X.................] duration: 4.196362669s trace: https://jaeger.dev.storj.io/trace/30cdf86753ce10d1 FAIL (2025-12-10 23:29:55 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 4.196391463s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [.....................................................XXXXX.X] duration: 1m0.241547692s trace: https://jaeger.dev.storj.io/trace/694c44ddbcba2d4d FAIL (2025-12-10 23:29:21 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== ML4MWMM52Q6IC7PQG5FQ, seg_pos={0 0}, upload=111768]: Uploading piece... Dec 10 23:28:22.535497558 (127h59m22.406580593s) [noise=true , address=77.90.31.10:4527 , piece_id=5727BK5KITJRUTTA5MDKNTKDJGH2CZMN2JQVJXFOTDN77L2CYQYQ, seg_pos={0 0}, upload=111768]: Uploading piece... Dec 10 23:28:22.536272273 (127h59m22.407355318s) [noise=true , address=77.90.31.12:4621 , piece_id=KTLDOHVBIQTFLM2V7ESCWWAGDDDTID7SFAJLRO5YLTQFITO6FAWQ, seg_pos={0 0}, upload=111768]: Uploading piece... Dec 10 23:28:22.536400147 (127h59m22.407483192s) [noise=true , address=77.90.31.21:4908 , piece_id=NDEACDEE73RGKN4HQK5NWT7P7DUR2OA4NXZ537HSJ7W36SVL3D2A, seg_pos={0 0}, upload=111768]: Uploading piece... Dec 10 23:28:22.538213800 (127h59m22.409296835s) [noise=true , address=77.90.31.13:4648 , piece_id=SW6M7OODP4ARI6JQFK33DHWJXW4GAICPLMOC2IDDU4Z7SR5N4XKQ, seg_pos={0 0}, upload=111768]: Uploading piece... Dec 10 23:28:22.538878927 (127h59m22.409961962s) [noise=true , address=77.90.31.11:4587 , piece_id=C5ARHUAQ5OQRFYAYCJSSJ52BJUSEYBHHFKG5MWK4VZW76PA4YRGA, seg_pos={0 0}, upload=111768]: Uploading piece... Dec 10 23:28:22.539789610 (127h59m22.410872646s) [noise=true , address=45.91.116.5:4317 , piece_id=MXAOIYSINMNMU2G65DZ54MH2S3UI7GI4FE7AT4EEE2DGMWDDDR7Q, seg_pos={0 0}, upload=111768]: Uploading piece... Dec 10 23:28:22.541981043 (127h59m22.413064078s) [noise=true , address=145.79.227.16:4918 , piece_id=T2JQ7EETW2NE62FH75RDI3RUPZAJYV4PTNJ5WUYTEZVBADYBFWXA, seg_pos={0 0}, upload=111768]: Uploading piece... Dec 10 23:28:22.543683766 (127h59m22.414766801s) [noise=true , address=77.90.31.14:4667 , piece_id=5CTOF3QZE4BTA2LJEZPCYB5GQZXKKEYTHG4PHZ4XLGGPXM4HUX4Q, seg_pos={0 0}, upload=111768]: Uploading piece... Dec 10 23:28:22.544038070 (127h59m22.415121105s) [noise=true , address=77.90.31.17:4777 , piece_id=NVCMRFZ47NA642C7T6LPIH7AD4V4Z3ZSNWI5TV4N4TT3WRKULJ2A, seg_pos={0 0}, upload=111768]: Uploading piece... Dec 10 23:28:22.545238866 (127h59m22.416321901s) [noise=true , address=45.91.116.13:4177 , piece_id=34GK3CSUMZBHETWVMRQ3L7VAL6ED476J7HJ5HI3MDFD7GQTPLS7Q, seg_pos={0 0}, upload=111768]: Uploading piece... Dec 10 23:28:22.545503901 (127h59m22.416586936s) [noise=true , address=77.90.31.18:4832 , piece_id=EM6PLOYQTX7RCY7WATQ2TAPQNX5U56ALDUVBENEE22Y6DQKQ6OLQ, seg_pos={0 0}, upload=111768]: Uploading piece... Dec 10 23:28:22.547149505 (127h59m22.41823254s) [noise=true , address=145.79.227.39:4017 , piece_id=XYMQU6EPOOJ2NQYF7OKREAGJ3SXJIOPH5BKVO6AGRJ3CHORLL7LQ, seg_pos={0 0}, upload=111768]: Uploading piece... Dec 10 23:28:22.547427053 (127h59m22.418510098s) [noise=true , address=145.79.227.34:4913 , piece_id=K4XFLJVLM6A4NUMUVZOENIJGOA2GRAD3A2HVFJYB3HBFM5BBQ7AA, seg_pos={0 0}, upload=111768]: Uploading piece... Dec 10 23:28:22.548674539 (127h59m22.419757584s) [noise=true , address=45.91.116.12:4235 , piece_id=VSMTBBMT3GKXSSX6AVXLLAZURPHE7RHH27H6OG6235P6WLXX7Q2Q, seg_pos={0 0}, upload=111768]: Uploading piece... Dec 10 23:28:22.548920186 (127h59m22.420003231s) [noise=true , address=64.49.13.169:4710 , piece_id=NWES5JKII7QYCWI2RVJUYHWDVXGSWBLZKP7CJRRM7AYV5CMFJA4A, seg_pos={0 0}, upload=111768]: Uploading piece... Dec 10 23:28:22.549563551 (127h59m22.420646596s) [noise=true , address=145.79.227.12:4780 , piece_id=73GC4MU55CZGOTADHABKUHOHJDPYGXILIYBFEE7A35555A5AF6MQ, seg_pos={0 0}, upload=111768]: Uploading piece... Dec 10 23:28:22.553166262 (127h59m22.424249307s) [noise=true , address=5.231.73.2:4024 , piece_id=GVED2HFTYTQSIXSASRI63FF6KKATULGAHUFK4G3AALCAP56RYY2Q, seg_pos={0 0}, upload=111768]: Uploading piece... Dec 10 23:28:24.000605750 (127h59m23.871688795s) [upload=111768]: Next returned nil segment Dec 10 23:28:24.000628273 (127h59m23.871711308s) [upload=111768]: Waiting for error group managing segments... Dec 10 23:28:24.403448383 (127h59m24.274531418s) [noise=true , address=145.79.227.34:4913 , piece_id=K4XFLJVLM6A4NUMUVZOENIJGOA2GRAD3A2HVFJYB3HBFM5BBQ7AA, seg_pos={0 0}, upload=111768]: Done uploading piece. err: <nil> Dec 10 23:28:24.407185840 (127h59m24.278268885s) [noise=true , address=145.79.227.39:4017 , piece_id=XYMQU6EPOOJ2NQYF7OKREAGJ3SXJIOPH5BKVO6AGRJ3CHORLL7LQ, seg_pos={0 0}, upload=111768]: Done uploading piece. err: <nil> Dec 10 23:28:24.415324026 (127h59m24.286407071s) [noise=true , address=145.79.227.12:4780 , piece_id=73GC4MU55CZGOTADHABKUHOHJDPYGXILIYBFEE7A35555A5AF6MQ, seg_pos={0 0}, upload=111768]: Done uploading piece. err: <nil> Dec 10 23:28:24.485287637 (127h59m24.356370682s) [noise=true , address=5.231.73.2:4024 , piece_id=GVED2HFTYTQSIXSASRI63FF6KKATULGAHUFK4G3AALCAP56RYY2Q, seg_pos={0 0}, upload=111768]: Done uploading piece. err: <nil> Dec 10 23:28:24.504646960 (127h59m24.375730005s) [noise=true , address=45.91.116.12:4235 , piece_id=VSMTBBMT3GKXSSX6AVXLLAZURPHE7RHH27H6OG6235P6WLXX7Q2Q, seg_pos={0 0}, upload=111768]: Done uploading piece. err: <nil> Dec 10 23:28:24.543602297 (127h59m24.414685342s) [noise=true , address=45.91.116.13:4177 , piece_id=34GK3CSUMZBHETWVMRQ3L7VAL6ED476J7HJ5HI3MDFD7GQTPLS7Q, seg_pos={0 0}, upload=111768]: Done uploading piece. err: <nil> Dec 10 23:28:24.554780332 (127h59m24.425863367s) [noise=true , address=45.91.116.8:4074 , piece_id=N7RT4LBAI7DX5745QKK5FA5FXG3774RGML4MWMM52Q6IC7PQG5FQ, seg_pos={0 0}, upload=111768]: Done uploading piece. err: <nil> Dec 10 23:28:24.593311150 (127h59m24.464394205s) [noise=true , address=45.91.116.5:4317 , piece_id=MXAOIYSINMNMU2G65DZ54MH2S3UI7GI4FE7AT4EEE2DGMWDDDR7Q, seg_pos={0 0}, upload=111768]: Done uploading piece. err: <nil> Dec 10 23:28:25.457088080 (127h59m25.328171115s) [noise=true , address=145.79.227.16:4918 , piece_id=T2JQ7EETW2NE62FH75RDI3RUPZAJYV4PTNJ5WUYTEZVBADYBFWXA, seg_pos={0 0}, upload=111768]: Done uploading piece. err: <nil> Dec 10 23:28:38.507797045 (127h59m38.37888008s) [noise=true , address=64.49.13.169:4710 , piece_id=NWES5JKII7QYCWI2RVJUYHWDVXGSWBLZKP7CJRRM7AYV5CMFJA4A, seg_pos={0 0}, upload=111768]: Done uploading piece. err: <nil> Dec 10 23:29:21.602767860 (128h0m21.473850895s) [noise=true , address=77.90.31.21:4908 , piece_id=NDEACDEE73RGKN4HQK5NWT7P7DUR2OA4NXZ537HSJ7W36SVL3D2A, seg_pos={0 0}, upload=111768]: Done uploading piece. err: ecclient: upload failed (node:13hCBBV3cxyTWCAuG8QGGAoJRb6S8Ay23HpXYyBgRX4HZSSpga, address:77.90.31.21:4908): protocol: rpc: dial tcp 77.90.31.21:4908: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:29:21.602961778 (128h0m21.474044823s) [noise=true , address=77.90.31.13:4648 , piece_id=SW6M7OODP4ARI6JQFK33DHWJXW4GAICPLMOC2IDDU4Z7SR5N4XKQ, seg_pos={0 0}, upload=111768]: Done uploading piece. err: ecclient: upload failed (node:1niTsTaHjA6LoMMrkEsSnLcRDhZtVAjxsbXUXVeXzhUp3FPbWh, address:77.90.31.13:4648): protocol: rpc: dial tcp 77.90.31.13:4648: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:29:21.603095373 (128h0m21.474178408s) [noise=true , address=77.90.31.17:4777 , piece_id=NVCMRFZ47NA642C7T6LPIH7AD4V4Z3ZSNWI5TV4N4TT3WRKULJ2A, seg_pos={0 0}, upload=111768]: Done uploading piece. err: ecclient: upload failed (node:1DjhLCRQRkNECPQg8HG8kZcBn8jJmBZpNhdywxEK7ZUn1v19wM, address:77.90.31.17:4777): protocol: write tcp 188.166.225.170:46606->77.90.31.17:4777: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:29:21.603262161 (128h0m21.474345196s) [noise=true , address=77.90.31.10:4527 , piece_id=5727BK5KITJRUTTA5MDKNTKDJGH2CZMN2JQVJXFOTDN77L2CYQYQ, seg_pos={0 0}, upload=111768]: Done uploading piece. err: ecclient: upload failed (node:1jEyiUUU9NCVFMU8frNggxTTYBMkehsqN4GLsFRkrxubdobGY2, address:77.90.31.10:4527): protocol: rpc: dial tcp 77.90.31.10:4527: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:29:21.603425432 (128h0m21.474508477s) [noise=true , address=77.90.31.11:4587 , piece_id=C5ARHUAQ5OQRFYAYCJSSJ52BJUSEYBHHFKG5MWK4VZW76PA4YRGA, seg_pos={0 0}, upload=111768]: Done uploading piece. err: ecclient: upload failed (node:1twkDDcpRrmR5bFhjGKKccMxC4Y35xh7tLaFxawhBmZkVWh3Bb, address:77.90.31.11:4587): protocol: rpc: dial tcp 77.90.31.11:4587: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:29:21.603608270 (128h0m21.474691305s) [noise=true , address=77.90.31.18:4832 , piece_id=EM6PLOYQTX7RCY7WATQ2TAPQNX5U56ALDUVBENEE22Y6DQKQ6OLQ, seg_pos={0 0}, upload=111768]: Done uploading piece. err: ecclient: upload failed (node:12QnZ3q9oe4zPuLpS3V1DCBhyaU1umS5B7bpuAqPkaaY96ErWbo, address:77.90.31.18:4832): protocol: rpc: dial tcp 77.90.31.18:4832: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:29:21.603781961 (128h0m21.474865006s) [noise=true , address=77.90.31.12:4621 , piece_id=KTLDOHVBIQTFLM2V7ESCWWAGDDDTID7SFAJLRO5YLTQFITO6FAWQ, seg_pos={0 0}, upload=111768]: Done uploading piece. err: ecclient: upload failed (node:12oSWAY52TJGVyZk4HTfifZBHQv42QsrAsk1Ks9PH6FoEmAZiN3, address:77.90.31.12:4621): protocol: rpc: dial tcp 77.90.31.12:4621: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:29:21.603909202 (128h0m21.474992247s) [noise=true , address=77.90.31.14:4667 , piece_id=5CTOF3QZE4BTA2LJEZPCYB5GQZXKKEYTHG4PHZ4XLGGPXM4HUX4Q, seg_pos={0 0}, upload=111768]: Done uploading piece. err: ecclient: upload failed (node:12R5gJFhsmocYPNZf4yNnYqJwcwAKt9yiHmmqGMk3btzVjkFV4S, address:77.90.31.14:4667): protocol: write tcp 188.166.225.170:40562->77.90.31.14:4667: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:29:21.603932357 (128h0m21.475015392s) [seg_pos={0 0}, upload=111768]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Dec 10 23:29:21.841785776 (128h0m21.712868831s) [upload=111768]: Done uploading segments... Dec 10 23:29:21.841902629 (128h0m21.712985674s) [upload=111768]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 1m0.241570996s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.....X.....................................X................] duration: 5.001673338s trace: https://jaeger.dev.storj.io/trace/30cdf86753ce10d1 FAIL (2025-12-10 23:28:57 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 5.001699999s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [......X.....................................X...............] duration: 4.05991535s trace: https://jaeger.dev.storj.io/trace/30cdf86753ce10d1 FAIL (2025-12-10 23:28:03 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 4.059948794s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [......................................................XXXXX.] duration: 15.160417979s trace: https://jaeger.dev.storj.io/trace/115fe29e9ab3e68d FAIL (2025-12-10 23:27:47 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== d=UXWWHWWU7AX35PSEP3AYFED7Y2QP7B3JARYYYVNSWTVTIU6ZGOPQ, seg_pos={0 0}, upload=111742]: Uploading piece... Dec 10 23:26:32.952182573 (127h57m32.823265628s) [noise=true , address=5.231.73.2:4021 , piece_id=WQDAE2SIPIMEP3TMJD5HZLO42ZRKNUFFOLEVDXKFVGQFKERV7QWA, seg_pos={0 0}, upload=111742]: Uploading piece... Dec 10 23:26:32.952710037 (127h57m32.823793082s) [noise=true , address=77.90.31.18:4823 , piece_id=ZN7W6AD3322L33OOSVSFN57ZEJQPQOGDNBE4EIW3F3MVYFBZFQKA, seg_pos={0 0}, upload=111742]: Uploading piece... Dec 10 23:26:32.953214267 (127h57m32.824297312s) [noise=true , address=77.90.31.16:4742 , piece_id=MHDCS3KGQFMGOS4GLNYLUEYTDIOLOALDJ77Y2FDNBLBZRVSPJTSA, seg_pos={0 0}, upload=111742]: Uploading piece... Dec 10 23:26:32.953702667 (127h57m32.824785702s) [noise=true , address=45.91.116.5:4310 , piece_id=YJHBQ7TWK2S5VVR3HVKQJRTCT662G7G3LNRWKYJF33PKXX6HA3TQ, seg_pos={0 0}, upload=111742]: Uploading piece... Dec 10 23:26:32.954368676 (127h57m32.825451721s) [noise=true , address=77.90.31.10:4528 , piece_id=CAUAKJEELGMBZWOHM7IISESFFWO46NGDKAUD43SQX6GFHNO2XVWA, seg_pos={0 0}, upload=111742]: Uploading piece... Dec 10 23:26:32.954992042 (127h57m32.826075087s) [noise=true , address=45.91.116.10:4099 , piece_id=ZK76BUGZJRHMTA4VXLSFTVF5ZCNFH5SWMQT6N7ZJ4UHHOCIHWFHQ, seg_pos={0 0}, upload=111742]: Uploading piece... Dec 10 23:26:32.955641168 (127h57m32.826724213s) [noise=true , address=64.49.13.170:4727 , piece_id=HZANEUR7IRX6Y6TT2FK3YEWQKRMFHLVEG2ROWFUTOAHGAYSKHYIA, seg_pos={0 0}, upload=111742]: Uploading piece... Dec 10 23:26:32.956252643 (127h57m32.827335688s) [noise=true , address=45.91.116.9:4281 , piece_id=GL2CVT3IKSV67QONU6GSGLTLXQNWCNAWIY26GQGGI6M2A477D7EA, seg_pos={0 0}, upload=111742]: Uploading piece... Dec 10 23:26:32.958047620 (127h57m32.829130655s) [noise=true , address=45.91.116.11:4142 , piece_id=MEXBRDEPPVAVQBDKK5CPSFPTBQ6VPGG5NIZZ5VAUL56BK74BPJFA, seg_pos={0 0}, upload=111742]: Uploading piece... Dec 10 23:26:32.959956475 (127h57m32.83103951s) [noise=true , address=45.91.116.8:4074 , piece_id=WHFEIIGLF54SXQ5HL4KWD2UCYT62EQRHYCQLHZX4ZJQFTUUWHPEQ, seg_pos={0 0}, upload=111742]: Uploading piece... Dec 10 23:26:32.961936606 (127h57m32.833019641s) [noise=true , address=45.91.116.4:4956 , piece_id=JJ7EUKBUSK3M6EK5QJDI5OIUUOI2W4YAIUEHCVT6ZA6GFPLUY63Q, seg_pos={0 0}, upload=111742]: Uploading piece... Dec 10 23:26:32.963759467 (127h57m32.834842502s) [noise=true , address=77.90.31.12:4603 , piece_id=SNVY6RUKX3MQUTQADIHWRQNXBBWHY7IP2KJGVIJEGN5YR3AEPTGQ, seg_pos={0 0}, upload=111742]: Uploading piece... Dec 10 23:26:32.965489221 (127h57m32.836572256s) [noise=true , address=77.90.31.13:4642 , piece_id=CDFFGRBAWAPC7BWF7FADY7GB7BTRACL4PUM26TZ37A6KF5DV7QTA, seg_pos={0 0}, upload=111742]: Uploading piece... Dec 10 23:26:32.965964546 (127h57m32.837047581s) [noise=true , address=77.90.31.17:4782 , piece_id=5MPEOUTIEBBAZBDZPPJ6AWUIGW6PFSNCB6LBECKV4KUHKHSFNR5Q, seg_pos={0 0}, upload=111742]: Uploading piece... Dec 10 23:26:32.967522893 (127h57m32.838605938s) [noise=true , address=77.90.31.19:4856 , piece_id=P6KUKPJB3UNVHPPM4TRUNCB6K7EEXKFCWB23MUJZJN2QEENQ7DBQ, seg_pos={0 0}, upload=111742]: Uploading piece... Dec 10 23:26:32.968763285 (127h57m32.83984633s) [noise=true , address=193.149.164.143:4699 , piece_id=IEDCUPDO6VKWJKK4K3UL5RU6PXZKPVUVGN3ILQDA3JVRLD4KIO2Q, seg_pos={0 0}, upload=111742]: Uploading piece... Dec 10 23:26:32.970063581 (127h57m32.841146616s) [noise=true , address=77.90.31.11:4567 , piece_id=W6M4G6R26ZS565DUQ2LTDGIFMZB4RLFF3BXDZZWDO4YX3KTOML7Q, seg_pos={0 0}, upload=111742]: Uploading piece... Dec 10 23:26:34.284381205 (127h57m34.15546424s) [upload=111742]: Next returned nil segment Dec 10 23:26:34.284395763 (127h57m34.155478799s) [upload=111742]: Waiting for error group managing segments... Dec 10 23:26:34.626651800 (127h57m34.497734835s) [noise=true , address=64.49.13.170:4727 , piece_id=HZANEUR7IRX6Y6TT2FK3YEWQKRMFHLVEG2ROWFUTOAHGAYSKHYIA, seg_pos={0 0}, upload=111742]: Done uploading piece. err: <nil> Dec 10 23:26:34.648990646 (127h57m34.520073682s) [noise=true , address=193.149.164.143:4699 , piece_id=IEDCUPDO6VKWJKK4K3UL5RU6PXZKPVUVGN3ILQDA3JVRLD4KIO2Q, seg_pos={0 0}, upload=111742]: Done uploading piece. err: <nil> Dec 10 23:26:34.733043877 (127h57m34.604126912s) [noise=true , address=5.231.73.2:4021 , piece_id=WQDAE2SIPIMEP3TMJD5HZLO42ZRKNUFFOLEVDXKFVGQFKERV7QWA, seg_pos={0 0}, upload=111742]: Done uploading piece. err: <nil> Dec 10 23:26:34.942975157 (127h57m34.814058192s) [noise=true , address=45.91.116.10:4099 , piece_id=ZK76BUGZJRHMTA4VXLSFTVF5ZCNFH5SWMQT6N7ZJ4UHHOCIHWFHQ, seg_pos={0 0}, upload=111742]: Done uploading piece. err: <nil> Dec 10 23:26:34.968180061 (127h57m34.839263126s) [noise=true , address=45.91.116.6:4001 , piece_id=UXWWHWWU7AX35PSEP3AYFED7Y2QP7B3JARYYYVNSWTVTIU6ZGOPQ, seg_pos={0 0}, upload=111742]: Done uploading piece. err: <nil> Dec 10 23:26:34.976521474 (127h57m34.847604519s) [noise=true , address=45.91.116.4:4956 , piece_id=JJ7EUKBUSK3M6EK5QJDI5OIUUOI2W4YAIUEHCVT6ZA6GFPLUY63Q, seg_pos={0 0}, upload=111742]: Done uploading piece. err: <nil> Dec 10 23:26:34.980033612 (127h57m34.851116667s) [noise=true , address=45.91.116.9:4281 , piece_id=GL2CVT3IKSV67QONU6GSGLTLXQNWCNAWIY26GQGGI6M2A477D7EA, seg_pos={0 0}, upload=111742]: Done uploading piece. err: <nil> Dec 10 23:26:34.985095690 (127h57m34.856178725s) [noise=true , address=45.91.116.5:4310 , piece_id=YJHBQ7TWK2S5VVR3HVKQJRTCT662G7G3LNRWKYJF33PKXX6HA3TQ, seg_pos={0 0}, upload=111742]: Done uploading piece. err: <nil> Dec 10 23:26:35.000795695 (127h57m34.87187874s) [noise=true , address=45.91.116.8:4074 , piece_id=WHFEIIGLF54SXQ5HL4KWD2UCYT62EQRHYCQLHZX4ZJQFTUUWHPEQ, seg_pos={0 0}, upload=111742]: Done uploading piece. err: <nil> Dec 10 23:26:35.050622202 (127h57m34.921705247s) [noise=true , address=45.91.116.11:4142 , piece_id=MEXBRDEPPVAVQBDKK5CPSFPTBQ6VPGG5NIZZ5VAUL56BK74BPJFA, seg_pos={0 0}, upload=111742]: Done uploading piece. err: <nil> Dec 10 23:27:32.014986965 (127h58m31.88607001s) [noise=true , address=77.90.31.10:4528 , piece_id=CAUAKJEELGMBZWOHM7IISESFFWO46NGDKAUD43SQX6GFHNO2XVWA, seg_pos={0 0}, upload=111742]: Done uploading piece. err: ecclient: upload failed (node:15fkvAWei1E5Tn2BYkCavyZ8se6XwPJcmpr7aEsv8sbMtk1n3Q, address:77.90.31.10:4528): protocol: rpc: dial tcp 77.90.31.10:4528: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:27:32.015123585 (127h58m31.88620663s) [noise=true , address=77.90.31.19:4856 , piece_id=P6KUKPJB3UNVHPPM4TRUNCB6K7EEXKFCWB23MUJZJN2QEENQ7DBQ, seg_pos={0 0}, upload=111742]: Done uploading piece. err: ecclient: upload failed (node:1MntjJtcCDSRozKpesp6S2ZjYrMhxiQheDiPMPSCnavWBBGYt3, address:77.90.31.19:4856): protocol: rpc: dial tcp 77.90.31.19:4856: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:27:32.015253903 (127h58m31.886336949s) [noise=true , address=77.90.31.17:4782 , piece_id=5MPEOUTIEBBAZBDZPPJ6AWUIGW6PFSNCB6LBECKV4KUHKHSFNR5Q, seg_pos={0 0}, upload=111742]: Done uploading piece. err: ecclient: upload failed (node:12FtB4bKTHntdUQxF8G5o9nT73oS2xB7fgD44BkRfKDNFyZnKCk, address:77.90.31.17:4782): protocol: rpc: dial tcp 77.90.31.17:4782: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:27:32.015480525 (127h58m31.886563561s) [noise=true , address=77.90.31.12:4603 , piece_id=SNVY6RUKX3MQUTQADIHWRQNXBBWHY7IP2KJGVIJEGN5YR3AEPTGQ, seg_pos={0 0}, upload=111742]: Done uploading piece. err: ecclient: upload failed (node:126Sou5pnMm2eE2FJpv6WYX2VSqJvZZf9rFFJD6hFYDetRD28zu, address:77.90.31.12:4603): protocol: rpc: dial tcp 77.90.31.12:4603: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:27:32.015676578 (127h58m31.886759633s) [noise=true , address=77.90.31.13:4642 , piece_id=CDFFGRBAWAPC7BWF7FADY7GB7BTRACL4PUM26TZ37A6KF5DV7QTA, seg_pos={0 0}, upload=111742]: Done uploading piece. err: ecclient: upload failed (node:1xdfVQUEMLaGgw7hsBtHZVTKL9BTTxPeYve53efw8GPdSQDL9D, address:77.90.31.13:4642): protocol: rpc: dial tcp 77.90.31.13:4642: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:27:32.015775186 (127h58m31.886858231s) [noise=true , address=77.90.31.11:4567 , piece_id=W6M4G6R26ZS565DUQ2LTDGIFMZB4RLFF3BXDZZWDO4YX3KTOML7Q, seg_pos={0 0}, upload=111742]: Done uploading piece. err: ecclient: upload failed (node:12HJVmEuxjrwWCcdwxWYFC5fTULds1kiRH82DiRWipWhRT1Qwdk, address:77.90.31.11:4567): protocol: write tcp 188.166.225.170:58100->77.90.31.11:4567: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:27:32.015871289 (127h58m31.886954324s) [noise=true , address=77.90.31.18:4823 , piece_id=ZN7W6AD3322L33OOSVSFN57ZEJQPQOGDNBE4EIW3F3MVYFBZFQKA, seg_pos={0 0}, upload=111742]: Done uploading piece. err: ecclient: upload failed (node:12uUCUDKZDwj25UcEKQ4WiTuQFMRbWw2oymGAdUdMX8wcUSd7jS, address:77.90.31.18:4823): protocol: rpc: dial tcp 77.90.31.18:4823: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:27:32.016026164 (127h58m31.887109209s) [noise=true , address=77.90.31.16:4742 , piece_id=MHDCS3KGQFMGOS4GLNYLUEYTDIOLOALDJ77Y2FDNBLBZRVSPJTSA, seg_pos={0 0}, upload=111742]: Done uploading piece. err: ecclient: upload failed (node:1me3JXQFLGyr6REuWPk7f7arC5d72HcDbexXuez45dYBKqVA3k, address:77.90.31.16:4742): protocol: rpc: dial tcp 77.90.31.16:4742: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:27:32.016084355 (127h58m31.8871674s) [seg_pos={0 0}, upload=111742]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Dec 10 23:27:32.257445673 (127h58m32.128528718s) [upload=111742]: Done uploading segments... Dec 10 23:27:32.257488134 (127h58m32.128571189s) [upload=111742]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 15.16043404s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [.......................................................XXXXX] duration: 1m0.244095979s trace: https://jaeger.dev.storj.io/trace/115fe29e9ab3e68d FAIL (2025-12-10 23:27:32 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== d=UXWWHWWU7AX35PSEP3AYFED7Y2QP7B3JARYYYVNSWTVTIU6ZGOPQ, seg_pos={0 0}, upload=111742]: Uploading piece... Dec 10 23:26:32.952182573 (127h57m32.823265628s) [noise=true , address=5.231.73.2:4021 , piece_id=WQDAE2SIPIMEP3TMJD5HZLO42ZRKNUFFOLEVDXKFVGQFKERV7QWA, seg_pos={0 0}, upload=111742]: Uploading piece... Dec 10 23:26:32.952710037 (127h57m32.823793082s) [noise=true , address=77.90.31.18:4823 , piece_id=ZN7W6AD3322L33OOSVSFN57ZEJQPQOGDNBE4EIW3F3MVYFBZFQKA, seg_pos={0 0}, upload=111742]: Uploading piece... Dec 10 23:26:32.953214267 (127h57m32.824297312s) [noise=true , address=77.90.31.16:4742 , piece_id=MHDCS3KGQFMGOS4GLNYLUEYTDIOLOALDJ77Y2FDNBLBZRVSPJTSA, seg_pos={0 0}, upload=111742]: Uploading piece... Dec 10 23:26:32.953702667 (127h57m32.824785702s) [noise=true , address=45.91.116.5:4310 , piece_id=YJHBQ7TWK2S5VVR3HVKQJRTCT662G7G3LNRWKYJF33PKXX6HA3TQ, seg_pos={0 0}, upload=111742]: Uploading piece... Dec 10 23:26:32.954368676 (127h57m32.825451721s) [noise=true , address=77.90.31.10:4528 , piece_id=CAUAKJEELGMBZWOHM7IISESFFWO46NGDKAUD43SQX6GFHNO2XVWA, seg_pos={0 0}, upload=111742]: Uploading piece... Dec 10 23:26:32.954992042 (127h57m32.826075087s) [noise=true , address=45.91.116.10:4099 , piece_id=ZK76BUGZJRHMTA4VXLSFTVF5ZCNFH5SWMQT6N7ZJ4UHHOCIHWFHQ, seg_pos={0 0}, upload=111742]: Uploading piece... Dec 10 23:26:32.955641168 (127h57m32.826724213s) [noise=true , address=64.49.13.170:4727 , piece_id=HZANEUR7IRX6Y6TT2FK3YEWQKRMFHLVEG2ROWFUTOAHGAYSKHYIA, seg_pos={0 0}, upload=111742]: Uploading piece... Dec 10 23:26:32.956252643 (127h57m32.827335688s) [noise=true , address=45.91.116.9:4281 , piece_id=GL2CVT3IKSV67QONU6GSGLTLXQNWCNAWIY26GQGGI6M2A477D7EA, seg_pos={0 0}, upload=111742]: Uploading piece... Dec 10 23:26:32.958047620 (127h57m32.829130655s) [noise=true , address=45.91.116.11:4142 , piece_id=MEXBRDEPPVAVQBDKK5CPSFPTBQ6VPGG5NIZZ5VAUL56BK74BPJFA, seg_pos={0 0}, upload=111742]: Uploading piece... Dec 10 23:26:32.959956475 (127h57m32.83103951s) [noise=true , address=45.91.116.8:4074 , piece_id=WHFEIIGLF54SXQ5HL4KWD2UCYT62EQRHYCQLHZX4ZJQFTUUWHPEQ, seg_pos={0 0}, upload=111742]: Uploading piece... Dec 10 23:26:32.961936606 (127h57m32.833019641s) [noise=true , address=45.91.116.4:4956 , piece_id=JJ7EUKBUSK3M6EK5QJDI5OIUUOI2W4YAIUEHCVT6ZA6GFPLUY63Q, seg_pos={0 0}, upload=111742]: Uploading piece... Dec 10 23:26:32.963759467 (127h57m32.834842502s) [noise=true , address=77.90.31.12:4603 , piece_id=SNVY6RUKX3MQUTQADIHWRQNXBBWHY7IP2KJGVIJEGN5YR3AEPTGQ, seg_pos={0 0}, upload=111742]: Uploading piece... Dec 10 23:26:32.965489221 (127h57m32.836572256s) [noise=true , address=77.90.31.13:4642 , piece_id=CDFFGRBAWAPC7BWF7FADY7GB7BTRACL4PUM26TZ37A6KF5DV7QTA, seg_pos={0 0}, upload=111742]: Uploading piece... Dec 10 23:26:32.965964546 (127h57m32.837047581s) [noise=true , address=77.90.31.17:4782 , piece_id=5MPEOUTIEBBAZBDZPPJ6AWUIGW6PFSNCB6LBECKV4KUHKHSFNR5Q, seg_pos={0 0}, upload=111742]: Uploading piece... Dec 10 23:26:32.967522893 (127h57m32.838605938s) [noise=true , address=77.90.31.19:4856 , piece_id=P6KUKPJB3UNVHPPM4TRUNCB6K7EEXKFCWB23MUJZJN2QEENQ7DBQ, seg_pos={0 0}, upload=111742]: Uploading piece... Dec 10 23:26:32.968763285 (127h57m32.83984633s) [noise=true , address=193.149.164.143:4699 , piece_id=IEDCUPDO6VKWJKK4K3UL5RU6PXZKPVUVGN3ILQDA3JVRLD4KIO2Q, seg_pos={0 0}, upload=111742]: Uploading piece... Dec 10 23:26:32.970063581 (127h57m32.841146616s) [noise=true , address=77.90.31.11:4567 , piece_id=W6M4G6R26ZS565DUQ2LTDGIFMZB4RLFF3BXDZZWDO4YX3KTOML7Q, seg_pos={0 0}, upload=111742]: Uploading piece... Dec 10 23:26:34.284381205 (127h57m34.15546424s) [upload=111742]: Next returned nil segment Dec 10 23:26:34.284395763 (127h57m34.155478799s) [upload=111742]: Waiting for error group managing segments... Dec 10 23:26:34.626651800 (127h57m34.497734835s) [noise=true , address=64.49.13.170:4727 , piece_id=HZANEUR7IRX6Y6TT2FK3YEWQKRMFHLVEG2ROWFUTOAHGAYSKHYIA, seg_pos={0 0}, upload=111742]: Done uploading piece. err: <nil> Dec 10 23:26:34.648990646 (127h57m34.520073682s) [noise=true , address=193.149.164.143:4699 , piece_id=IEDCUPDO6VKWJKK4K3UL5RU6PXZKPVUVGN3ILQDA3JVRLD4KIO2Q, seg_pos={0 0}, upload=111742]: Done uploading piece. err: <nil> Dec 10 23:26:34.733043877 (127h57m34.604126912s) [noise=true , address=5.231.73.2:4021 , piece_id=WQDAE2SIPIMEP3TMJD5HZLO42ZRKNUFFOLEVDXKFVGQFKERV7QWA, seg_pos={0 0}, upload=111742]: Done uploading piece. err: <nil> Dec 10 23:26:34.942975157 (127h57m34.814058192s) [noise=true , address=45.91.116.10:4099 , piece_id=ZK76BUGZJRHMTA4VXLSFTVF5ZCNFH5SWMQT6N7ZJ4UHHOCIHWFHQ, seg_pos={0 0}, upload=111742]: Done uploading piece. err: <nil> Dec 10 23:26:34.968180061 (127h57m34.839263126s) [noise=true , address=45.91.116.6:4001 , piece_id=UXWWHWWU7AX35PSEP3AYFED7Y2QP7B3JARYYYVNSWTVTIU6ZGOPQ, seg_pos={0 0}, upload=111742]: Done uploading piece. err: <nil> Dec 10 23:26:34.976521474 (127h57m34.847604519s) [noise=true , address=45.91.116.4:4956 , piece_id=JJ7EUKBUSK3M6EK5QJDI5OIUUOI2W4YAIUEHCVT6ZA6GFPLUY63Q, seg_pos={0 0}, upload=111742]: Done uploading piece. err: <nil> Dec 10 23:26:34.980033612 (127h57m34.851116667s) [noise=true , address=45.91.116.9:4281 , piece_id=GL2CVT3IKSV67QONU6GSGLTLXQNWCNAWIY26GQGGI6M2A477D7EA, seg_pos={0 0}, upload=111742]: Done uploading piece. err: <nil> Dec 10 23:26:34.985095690 (127h57m34.856178725s) [noise=true , address=45.91.116.5:4310 , piece_id=YJHBQ7TWK2S5VVR3HVKQJRTCT662G7G3LNRWKYJF33PKXX6HA3TQ, seg_pos={0 0}, upload=111742]: Done uploading piece. err: <nil> Dec 10 23:26:35.000795695 (127h57m34.87187874s) [noise=true , address=45.91.116.8:4074 , piece_id=WHFEIIGLF54SXQ5HL4KWD2UCYT62EQRHYCQLHZX4ZJQFTUUWHPEQ, seg_pos={0 0}, upload=111742]: Done uploading piece. err: <nil> Dec 10 23:26:35.050622202 (127h57m34.921705247s) [noise=true , address=45.91.116.11:4142 , piece_id=MEXBRDEPPVAVQBDKK5CPSFPTBQ6VPGG5NIZZ5VAUL56BK74BPJFA, seg_pos={0 0}, upload=111742]: Done uploading piece. err: <nil> Dec 10 23:27:32.014986965 (127h58m31.88607001s) [noise=true , address=77.90.31.10:4528 , piece_id=CAUAKJEELGMBZWOHM7IISESFFWO46NGDKAUD43SQX6GFHNO2XVWA, seg_pos={0 0}, upload=111742]: Done uploading piece. err: ecclient: upload failed (node:15fkvAWei1E5Tn2BYkCavyZ8se6XwPJcmpr7aEsv8sbMtk1n3Q, address:77.90.31.10:4528): protocol: rpc: dial tcp 77.90.31.10:4528: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:27:32.015123585 (127h58m31.88620663s) [noise=true , address=77.90.31.19:4856 , piece_id=P6KUKPJB3UNVHPPM4TRUNCB6K7EEXKFCWB23MUJZJN2QEENQ7DBQ, seg_pos={0 0}, upload=111742]: Done uploading piece. err: ecclient: upload failed (node:1MntjJtcCDSRozKpesp6S2ZjYrMhxiQheDiPMPSCnavWBBGYt3, address:77.90.31.19:4856): protocol: rpc: dial tcp 77.90.31.19:4856: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:27:32.015253903 (127h58m31.886336949s) [noise=true , address=77.90.31.17:4782 , piece_id=5MPEOUTIEBBAZBDZPPJ6AWUIGW6PFSNCB6LBECKV4KUHKHSFNR5Q, seg_pos={0 0}, upload=111742]: Done uploading piece. err: ecclient: upload failed (node:12FtB4bKTHntdUQxF8G5o9nT73oS2xB7fgD44BkRfKDNFyZnKCk, address:77.90.31.17:4782): protocol: rpc: dial tcp 77.90.31.17:4782: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:27:32.015480525 (127h58m31.886563561s) [noise=true , address=77.90.31.12:4603 , piece_id=SNVY6RUKX3MQUTQADIHWRQNXBBWHY7IP2KJGVIJEGN5YR3AEPTGQ, seg_pos={0 0}, upload=111742]: Done uploading piece. err: ecclient: upload failed (node:126Sou5pnMm2eE2FJpv6WYX2VSqJvZZf9rFFJD6hFYDetRD28zu, address:77.90.31.12:4603): protocol: rpc: dial tcp 77.90.31.12:4603: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:27:32.015676578 (127h58m31.886759633s) [noise=true , address=77.90.31.13:4642 , piece_id=CDFFGRBAWAPC7BWF7FADY7GB7BTRACL4PUM26TZ37A6KF5DV7QTA, seg_pos={0 0}, upload=111742]: Done uploading piece. err: ecclient: upload failed (node:1xdfVQUEMLaGgw7hsBtHZVTKL9BTTxPeYve53efw8GPdSQDL9D, address:77.90.31.13:4642): protocol: rpc: dial tcp 77.90.31.13:4642: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:27:32.015775186 (127h58m31.886858231s) [noise=true , address=77.90.31.11:4567 , piece_id=W6M4G6R26ZS565DUQ2LTDGIFMZB4RLFF3BXDZZWDO4YX3KTOML7Q, seg_pos={0 0}, upload=111742]: Done uploading piece. err: ecclient: upload failed (node:12HJVmEuxjrwWCcdwxWYFC5fTULds1kiRH82DiRWipWhRT1Qwdk, address:77.90.31.11:4567): protocol: write tcp 188.166.225.170:58100->77.90.31.11:4567: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:27:32.015871289 (127h58m31.886954324s) [noise=true , address=77.90.31.18:4823 , piece_id=ZN7W6AD3322L33OOSVSFN57ZEJQPQOGDNBE4EIW3F3MVYFBZFQKA, seg_pos={0 0}, upload=111742]: Done uploading piece. err: ecclient: upload failed (node:12uUCUDKZDwj25UcEKQ4WiTuQFMRbWw2oymGAdUdMX8wcUSd7jS, address:77.90.31.18:4823): protocol: rpc: dial tcp 77.90.31.18:4823: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:27:32.016026164 (127h58m31.887109209s) [noise=true , address=77.90.31.16:4742 , piece_id=MHDCS3KGQFMGOS4GLNYLUEYTDIOLOALDJ77Y2FDNBLBZRVSPJTSA, seg_pos={0 0}, upload=111742]: Done uploading piece. err: ecclient: upload failed (node:1me3JXQFLGyr6REuWPk7f7arC5d72HcDbexXuez45dYBKqVA3k, address:77.90.31.16:4742): protocol: rpc: dial tcp 77.90.31.16:4742: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:27:32.016084355 (127h58m31.8871674s) [seg_pos={0 0}, upload=111742]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Dec 10 23:27:32.257445673 (127h58m32.128528718s) [upload=111742]: Done uploading segments... Dec 10 23:27:32.257488134 (127h58m32.128571189s) [upload=111742]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 1m0.244110987s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.......X.....................................X..............] duration: 4.137548918s trace: https://jaeger.dev.storj.io/trace/30cdf86753ce10d1 FAIL (2025-12-10 23:26:56 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 4.137593633s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [........................................................XXXX] duration: 1m0.248525213s trace: https://jaeger.dev.storj.io/trace/514745f843418f63 FAIL (2025-12-10 23:26:31 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== 4469640 (127h56m32.345552675s) [noise=true , address=145.79.227.39:4021 , piece_id=QPVHQJKRBZI6VAEWZLHKT3JQGTALKT6MPGGPR4OPPMXALDM3SPJA, seg_pos={0 0}, upload=111727]: Uploading piece... Dec 10 23:25:32.474993938 (127h56m32.346076984s) [noise=true , address=5.231.73.5:4010 , piece_id=2P46UCPY3QOVYMPOLHUATVDYZDINUT2VVVMO7JR2CSI5CSRJVNQA, seg_pos={0 0}, upload=111727]: Uploading piece... Dec 10 23:25:32.475404440 (127h56m32.346487485s) [noise=true , address=45.91.116.11:4127 , piece_id=TRVVABG6NTUSBSKIDSZ42PPVGJSXA5CKJA5CMPJKHPK46WW4B3LQ, seg_pos={0 0}, upload=111727]: Uploading piece... Dec 10 23:25:32.475828958 (127h56m32.346912004s) [noise=true , address=145.79.227.3:4180 , piece_id=AKZM3RT6W4JQZMIGNGKKPESM7YTARMHC2AO6RTSAS23BHXFAD3BQ, seg_pos={0 0}, upload=111727]: Uploading piece... Dec 10 23:25:32.476246103 (127h56m32.347329148s) [noise=true , address=45.91.116.6:4988 , piece_id=QCJ4BKOWVCQCS72VFEUVFLSQFXYVNZESMD2GGOKWCGSF5NSPAGHQ, seg_pos={0 0}, upload=111727]: Uploading piece... Dec 10 23:25:32.476663427 (127h56m32.347746473s) [noise=true , address=77.90.31.15:4707 , piece_id=G5QK3ECXGYL2WKNOX4ECYDHV6XOSL627ZON2NAEURGZ6MUAIDCVA, seg_pos={0 0}, upload=111727]: Uploading piece... Dec 10 23:25:32.477118674 (127h56m32.348201719s) [noise=true , address=145.79.227.27:4703 , piece_id=JRXAAEB4DKC5B6G7JRHWFNSDQ426VNSQF2T7QEHD5ZDKRTPOTKGQ, seg_pos={0 0}, upload=111727]: Uploading piece... Dec 10 23:25:32.477787327 (127h56m32.348870372s) [noise=true , address=77.90.31.17:4791 , piece_id=NZDSGUUJUI4HFA5HQGLF4LWIQ7VEYOMRSXIGTY5Y2E5BPH2IMLAA, seg_pos={0 0}, upload=111727]: Uploading piece... Dec 10 23:25:32.478034960 (127h56m32.349118005s) [noise=true , address=77.90.31.18:4817 , piece_id=HPGJNGG67PTS3LYTWFR6UW4GZXNZ3HWB74HADS2GPNAPEEPWYXJQ, seg_pos={0 0}, upload=111727]: Uploading piece... Dec 10 23:25:32.478463465 (127h56m32.3495465s) [noise=true , address=5.231.73.2:4031 , piece_id=6BBEHOI5PM7SKKUO7VZPPMFNIGXSQBMQTAR7KLRNKXVHR6QQJSAQ, seg_pos={0 0}, upload=111727]: Uploading piece... Dec 10 23:25:32.478571892 (127h56m32.349654927s) [noise=true , address=77.90.31.10:4549 , piece_id=BSYIS4ZBIJZAHOGUP5ULH5UGFIS2G4GXJ45OQHA5HSZRUWYJ5CWQ, seg_pos={0 0}, upload=111727]: Uploading piece... Dec 10 23:25:32.478955542 (127h56m32.350038577s) [noise=true , address=77.90.31.21:4898 , piece_id=PCD2UOYH2UPI4QVHL6WTHZX4GF7A55AUG6XBIIFUGSYDR7CQN6FQ, seg_pos={0 0}, upload=111727]: Uploading piece... Dec 10 23:25:32.479020005 (127h56m32.35010305s) [noise=true , address=45.91.116.10:4104 , piece_id=EYOK22QT3LWXE7BJ4OVNMKTYBJNRSO6GNLLV3C5TMC4JUJI7U6XA, seg_pos={0 0}, upload=111727]: Uploading piece... Dec 10 23:25:32.479346897 (127h56m32.350429932s) [noise=true , address=45.91.116.12:4236 , piece_id=XWA626NTQVAVOWLX2C4V7J2EEPY3U4RZ54AKTIUQ5NM4Z27J64XA, seg_pos={0 0}, upload=111727]: Uploading piece... Dec 10 23:25:32.479753331 (127h56m32.350836376s) [noise=true , address=77.90.31.19:4871 , piece_id=2PGLVAVV2A5J6CXXKZRSEPY4ADFR57AUT4FPYU6BCZFWRBTGJR7Q, seg_pos={0 0}, upload=111727]: Uploading piece... Dec 10 23:25:32.480155066 (127h56m32.351238101s) [noise=true , address=77.90.31.12:4624 , piece_id=UXZOHOFBR7VPF4WRSXNXVGNVZ6XDCWGGSPXSF5AA3OI77RQCIKRQ, seg_pos={0 0}, upload=111727]: Uploading piece... Dec 10 23:25:32.480672561 (127h56m32.351755606s) [noise=true , address=77.90.31.10:4550 , piece_id=UYUEJ63F3V5SAIMDIMGWBGEGPH4NGVPBZTRLJXDQ7NVRXBKESWXA, seg_pos={0 0}, upload=111727]: Uploading piece... Dec 10 23:25:33.891208672 (127h56m33.762291707s) [upload=111727]: Next returned nil segment Dec 10 23:25:33.891226515 (127h56m33.76230955s) [upload=111727]: Waiting for error group managing segments... Dec 10 23:25:34.262874929 (127h56m34.133957974s) [noise=true , address=5.231.73.2:4031 , piece_id=6BBEHOI5PM7SKKUO7VZPPMFNIGXSQBMQTAR7KLRNKXVHR6QQJSAQ, seg_pos={0 0}, upload=111727]: Done uploading piece. err: <nil> Dec 10 23:25:34.343115268 (127h56m34.214198313s) [noise=true , address=145.79.227.39:4021 , piece_id=QPVHQJKRBZI6VAEWZLHKT3JQGTALKT6MPGGPR4OPPMXALDM3SPJA, seg_pos={0 0}, upload=111727]: Done uploading piece. err: <nil> Dec 10 23:25:34.345091401 (127h56m34.216174436s) [noise=true , address=145.79.227.27:4703 , piece_id=JRXAAEB4DKC5B6G7JRHWFNSDQ426VNSQF2T7QEHD5ZDKRTPOTKGQ, seg_pos={0 0}, upload=111727]: Done uploading piece. err: <nil> Dec 10 23:25:34.347542648 (127h56m34.218625693s) [noise=true , address=145.79.227.29:4944 , piece_id=EVJMR3WUHBJ7XODQEGQ3QQTWPX2NE5IAVLA3VV7YOT2XO2PPTXFQ, seg_pos={0 0}, upload=111727]: Done uploading piece. err: <nil> Dec 10 23:25:34.375505073 (127h56m34.246588108s) [noise=true , address=145.79.227.3:4180 , piece_id=AKZM3RT6W4JQZMIGNGKKPESM7YTARMHC2AO6RTSAS23BHXFAD3BQ, seg_pos={0 0}, upload=111727]: Done uploading piece. err: <nil> Dec 10 23:25:34.411514992 (127h56m34.282598037s) [noise=true , address=5.231.73.5:4010 , piece_id=2P46UCPY3QOVYMPOLHUATVDYZDINUT2VVVMO7JR2CSI5CSRJVNQA, seg_pos={0 0}, upload=111727]: Done uploading piece. err: <nil> Dec 10 23:25:34.507288348 (127h56m34.378371393s) [noise=true , address=45.91.116.11:4127 , piece_id=TRVVABG6NTUSBSKIDSZ42PPVGJSXA5CKJA5CMPJKHPK46WW4B3LQ, seg_pos={0 0}, upload=111727]: Done uploading piece. err: <nil> Dec 10 23:25:34.517556771 (127h56m34.388639816s) [noise=true , address=45.91.116.6:4988 , piece_id=QCJ4BKOWVCQCS72VFEUVFLSQFXYVNZESMD2GGOKWCGSF5NSPAGHQ, seg_pos={0 0}, upload=111727]: Done uploading piece. err: <nil> Dec 10 23:25:34.542085978 (127h56m34.413169033s) [noise=true , address=45.91.116.10:4104 , piece_id=EYOK22QT3LWXE7BJ4OVNMKTYBJNRSO6GNLLV3C5TMC4JUJI7U6XA, seg_pos={0 0}, upload=111727]: Done uploading piece. err: <nil> Dec 10 23:25:34.553342892 (127h56m34.424425928s) [noise=true , address=45.91.116.12:4236 , piece_id=XWA626NTQVAVOWLX2C4V7J2EEPY3U4RZ54AKTIUQ5NM4Z27J64XA, seg_pos={0 0}, upload=111727]: Done uploading piece. err: <nil> Dec 10 23:26:31.544403602 (127h57m31.415486637s) [noise=true , address=77.90.31.12:4624 , piece_id=UXZOHOFBR7VPF4WRSXNXVGNVZ6XDCWGGSPXSF5AA3OI77RQCIKRQ, seg_pos={0 0}, upload=111727]: Done uploading piece. err: ecclient: upload failed (node:1KxpgfHEQjqnumtfhQ2TEYCgZPFbUQVShfJA7dgNbnZXpncMnV, address:77.90.31.12:4624): protocol: rpc: dial tcp 77.90.31.12:4624: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:26:31.544601979 (127h57m31.415685025s) [noise=true , address=77.90.31.10:4550 , piece_id=UYUEJ63F3V5SAIMDIMGWBGEGPH4NGVPBZTRLJXDQ7NVRXBKESWXA, seg_pos={0 0}, upload=111727]: Done uploading piece. err: ecclient: upload failed (node:12nfBW1W3RvEGzRz2bYA1hGxMfvsxVR6WNn5oPUdtJU26KMXNE7, address:77.90.31.10:4550): protocol: rpc: dial tcp 77.90.31.10:4550: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:26:31.544894046 (127h57m31.415977101s) [noise=true , address=77.90.31.17:4791 , piece_id=NZDSGUUJUI4HFA5HQGLF4LWIQ7VEYOMRSXIGTY5Y2E5BPH2IMLAA, seg_pos={0 0}, upload=111727]: Done uploading piece. err: ecclient: upload failed (node:1bFEmP78qGhyHhXr3QNVptdiWQ4No4nrb31qj6DPgmJLcHYKiM, address:77.90.31.17:4791): protocol: rpc: dial tcp 77.90.31.17:4791: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:26:31.545195570 (127h57m31.416278615s) [noise=true , address=77.90.31.18:4817 , piece_id=HPGJNGG67PTS3LYTWFR6UW4GZXNZ3HWB74HADS2GPNAPEEPWYXJQ, seg_pos={0 0}, upload=111727]: Done uploading piece. err: ecclient: upload failed (node:12Qv5LBuANP9vS2vbsVNrJ7Mkc3SqXwTNCBzLvN955RHFKQVv5x, address:77.90.31.18:4817): protocol: write tcp 188.166.225.170:43138->77.90.31.18:4817: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:26:31.545442780 (127h57m31.416525825s) [noise=true , address=77.90.31.10:4549 , piece_id=BSYIS4ZBIJZAHOGUP5ULH5UGFIS2G4GXJ45OQHA5HSZRUWYJ5CWQ, seg_pos={0 0}, upload=111727]: Done uploading piece. err: ecclient: upload failed (node:1Uay43uEvzmHudZaGM1ZcEFbEdUXzQto5RtZNiQXgrqaRmuQoy, address:77.90.31.10:4549): protocol: write tcp 188.166.225.170:48204->77.90.31.10:4549: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:26:31.545477286 (127h57m31.416560331s) [noise=true , address=77.90.31.21:4898 , piece_id=PCD2UOYH2UPI4QVHL6WTHZX4GF7A55AUG6XBIIFUGSYDR7CQN6FQ, seg_pos={0 0}, upload=111727]: Done uploading piece. err: ecclient: upload failed (node:12Qi796rSqngMJNBiFji2g4eYRn5MsqCqhpGPdYPN2GN8yFGTUx, address:77.90.31.21:4898): protocol: write tcp 188.166.225.170:54676->77.90.31.21:4898: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:26:31.545656799 (127h57m31.416739844s) [noise=true , address=77.90.31.19:4871 , piece_id=2PGLVAVV2A5J6CXXKZRSEPY4ADFR57AUT4FPYU6BCZFWRBTGJR7Q, seg_pos={0 0}, upload=111727]: Done uploading piece. err: ecclient: upload failed (node:12hrM59APBvzYUrwWMvSUQVMSkzNPxaNFDWEZn2J96xYTdv4Ezm, address:77.90.31.19:4871): protocol: write tcp 188.166.225.170:48510->77.90.31.19:4871: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:26:31.545772439 (127h57m31.416855474s) [noise=true , address=77.90.31.15:4707 , piece_id=G5QK3ECXGYL2WKNOX4ECYDHV6XOSL627ZON2NAEURGZ6MUAIDCVA, seg_pos={0 0}, upload=111727]: Done uploading piece. err: ecclient: upload failed (node:1wPoPpMVDQM4YDQzYbfLTh7aneNVXFisUY6iZCHXG1Lmkj7UrL, address:77.90.31.15:4707): protocol: rpc: dial tcp 77.90.31.15:4707: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:26:31.545877308 (127h57m31.416960353s) [seg_pos={0 0}, upload=111727]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Dec 10 23:26:31.790974380 (127h57m31.662057415s) [upload=111727]: Done uploading segments... Dec 10 23:26:31.791025958 (127h57m31.662108993s) [upload=111727]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 1m0.248536895s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [........X.....................................X.............] duration: 4.064224518s trace: https://jaeger.dev.storj.io/trace/30cdf86753ce10d1 FAIL (2025-12-10 23:26:18 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 4.064266649s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)