Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 20 out of 60 is failed (threshold=6) [.......X.......X.....X.X.XX...XXXXX.XX..XX.XXXX.....X.......] duration: 3.981432974s trace: https://jaeger.dev.storj.io/trace/142285b66b9055cd FAIL (2025-12-18 14:45:16 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.981449675s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.................X....................X.....................] duration: 5.95774097s trace: https://jaeger.dev.storj.io/trace/2cbc07c217c30b3f FAIL (2025-12-18 14:44:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, Piece ID: O7IY6V6T4UILG3SXQ6K3SPXBVA2VLB22PWVQYIYORGLLYN6LE64A) context deadline exceeded; piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: I2SLWBEKFHZXWYTBZF73DICAZ4ZBXQYU6DES2QWLJMUCFZOCC7CA) context deadline exceeded ======LOGS====== ec 18 14:26:45.953557484 (310h57m45.824640529s) [upload=272678]: Starting upload Dec 18 14:26:45.953631205 (310h57m45.82471425s) [upload=272678]: Done starting upload Dec 18 14:26:45.953727147 (310h57m45.824810202s) [upload=272678]: Uploading segments... Dec 18 14:26:45.953920536 (310h57m45.825003581s) [upload=272678]: Got next segment. Inline: false Dec 18 14:26:45.953955743 (310h57m45.825038798s) [upload=272678]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 18 14:26:45.953963277 (310h57m45.825046322s) [upload=272678]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 18 14:26:46.201381593 (310h57m46.072464628s) [seg_pos={0 0}, upload=272678]: Begin upload segment... Dec 18 14:26:46.201430356 (310h57m46.072513391s) [seg_pos={0 0}, upload=272678]: Done begin upload segment. Dec 18 14:26:46.201523023 (310h57m46.072606088s) [noise=true , address=145.79.227.45:4231 , piece_id=V7B6RPI74SKRJQZCUWFHB2ECD46T3V224VKBL47NIIK5YH6PJCUQ, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.201534584 (310h57m46.072617619s) [noise=true , address=45.91.116.8:4048 , piece_id=BHQYLAJFKTBQ2WGY4XGRI4UOPTGB3SFEUVYZ7XWAC2BKA3ISU3WA, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.203237186 (310h57m46.074320231s) [noise=true , address=45.91.116.14:4190 , piece_id=I2SLWBEKFHZXWYTBZF73DICAZ4ZBXQYU6DES2QWLJMUCFZOCC7CA, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.203669961 (310h57m46.074752996s) [noise=true , address=145.79.227.36:4519 , piece_id=3E54JIZJCR55H36BEOLTDKWRMYA3WHNDPBSALSMNRY45O3BYO6JA, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.203970172 (310h57m46.075053207s) [noise=true , address=145.79.227.38:4978 , piece_id=FRWJ6Z3EH45YRMQVU235FJB75JWK7HOCLZXS7ASITOHLTL3YY5PQ, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.204250205 (310h57m46.07533324s) [noise=true , address=45.91.116.10:4084 , piece_id=UOTQVSUYETMA7UOEURUAT5RPX4QUO4FOXSLQY7UDUKJBCYCOH5BA, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.204534577 (310h57m46.075617612s) [noise=true , address=45.91.116.5:4299 , piece_id=SIZBVTWHVPXBL7OMRRQ467KFLRB56JNA5WDRNSXSNHPSV7UYG5LQ, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.204821995 (310h57m46.07590504s) [noise=true , address=145.79.227.29:4943 , piece_id=4DWWCYWOAU7NSESXDCONC6THXO3TTUYJMN7G6PTG2CP2OOOY2USA, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.206113864 (310h57m46.0771969s) [noise=true , address=77.90.31.11:4556 , piece_id=4GEBIFAQJYJX4KTPHTYUODQZ6ABTTPOA5SYOJBGUV7YDFIHHVO2Q, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.206151877 (310h57m46.077234912s) [noise=true , address=145.79.227.3:4150 , piece_id=XR5E35XW6KBFTREM2GFHARKAAIAINOWMLVYDKKBPFQBXJZOFCC2Q, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.207752204 (310h57m46.078835239s) [noise=true , address=145.79.227.39:4013 , piece_id=2SYWMQU5R7MCTH2E4X732AOVIRVPGUNNXLIN3UFCU7WF4BEJUACQ, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.210366552 (310h57m46.081449597s) [noise=true , address=145.79.227.35:4482 , piece_id=3EGA2CLZBLJWSSULQCBLEEHCMCJDE2FSOY4FCANAQXPEYTWIF53A, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.211613657 (310h57m46.082696692s) [noise=true , address=145.79.227.47:4266 , piece_id=4444BNJTIBUQWLNGTEK3IFD4TIWTCIBCWUHSYBMD3FGZSNDPWT3Q, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.212828561 (310h57m46.083911596s) [noise=true , address=145.79.227.41:4086 , piece_id=DFSNBYWUJ6Z65VTU5ME2IGEDB6KLMTAJM65RC44FUVYHTPEW4ZSA, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.215707031 (310h57m46.086790066s) [noise=true , address=45.91.116.13:4155 , piece_id=O7IY6V6T4UILG3SXQ6K3SPXBVA2VLB22PWVQYIYORGLLYN6LE64A, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.216889753 (310h57m46.087972798s) [noise=true , address=145.79.227.31:4797 , piece_id=F5WHLSRZSSZ4CRJRR2ZPYGGG67SHQ3QMAIHIC42NOVDW2UXARINQ, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:47.912125819 (310h57m47.783208865s) [upload=272678]: Next returned nil segment Dec 18 14:26:47.912143884 (310h57m47.783226929s) [upload=272678]: Waiting for error group managing segments... Dec 18 14:26:48.193212096 (310h57m48.064295131s) [noise=true , address=145.79.227.45:4231 , piece_id=V7B6RPI74SKRJQZCUWFHB2ECD46T3V224VKBL47NIIK5YH6PJCUQ, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.207387668 (310h57m48.078470703s) [noise=true , address=145.79.227.31:4797 , piece_id=F5WHLSRZSSZ4CRJRR2ZPYGGG67SHQ3QMAIHIC42NOVDW2UXARINQ, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.219064714 (310h57m48.090147759s) [noise=true , address=145.79.227.39:4013 , piece_id=2SYWMQU5R7MCTH2E4X732AOVIRVPGUNNXLIN3UFCU7WF4BEJUACQ, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.227887805 (310h57m48.09897085s) [noise=true , address=145.79.227.36:4519 , piece_id=3E54JIZJCR55H36BEOLTDKWRMYA3WHNDPBSALSMNRY45O3BYO6JA, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.229044407 (310h57m48.100127442s) [noise=true , address=145.79.227.41:4086 , piece_id=DFSNBYWUJ6Z65VTU5ME2IGEDB6KLMTAJM65RC44FUVYHTPEW4ZSA, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.234560812 (310h57m48.105643857s) [noise=true , address=145.79.227.47:4266 , piece_id=4444BNJTIBUQWLNGTEK3IFD4TIWTCIBCWUHSYBMD3FGZSNDPWT3Q, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.237048037 (310h57m48.108131072s) [noise=true , address=145.79.227.35:4482 , piece_id=3EGA2CLZBLJWSSULQCBLEEHCMCJDE2FSOY4FCANAQXPEYTWIF53A, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.267241984 (310h57m48.138325019s) [noise=true , address=145.79.227.29:4943 , piece_id=4DWWCYWOAU7NSESXDCONC6THXO3TTUYJMN7G6PTG2CP2OOOY2USA, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.288475576 (310h57m48.159558621s) [noise=true , address=145.79.227.3:4150 , piece_id=XR5E35XW6KBFTREM2GFHARKAAIAINOWMLVYDKKBPFQBXJZOFCC2Q, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.293478393 (310h57m48.164561458s) [noise=true , address=145.79.227.38:4978 , piece_id=FRWJ6Z3EH45YRMQVU235FJB75JWK7HOCLZXS7ASITOHLTL3YY5PQ, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.423759785 (310h57m48.29484282s) [noise=true , address=45.91.116.14:4190 , piece_id=I2SLWBEKFHZXWYTBZF73DICAZ4ZBXQYU6DES2QWLJMUCFZOCC7CA, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.485697347 (310h57m48.356780392s) [noise=true , address=45.91.116.10:4084 , piece_id=UOTQVSUYETMA7UOEURUAT5RPX4QUO4FOXSLQY7UDUKJBCYCOH5BA, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.540540711 (310h57m48.411623756s) [noise=true , address=45.91.116.13:4155 , piece_id=O7IY6V6T4UILG3SXQ6K3SPXBVA2VLB22PWVQYIYORGLLYN6LE64A, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.540575787 (310h57m48.411658832s) [seg_pos={0 0}, upload=272678]: Segment reached optimal threshold of 13 pieces. Dec 18 14:26:48.540831424 (310h57m48.411914459s) [noise=true , address=45.91.116.8:4048 , piece_id=BHQYLAJFKTBQ2WGY4XGRI4UOPTGB3SFEUVYZ7XWAC2BKA3ISU3WA, seg_pos={0 0}, upload=272678]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: write tcp 188.166.225.170:49328->45.91.116.8:4048: use of closed network connection; write tcp 188.166.225.170:49328->45.91.116.8:4048: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49328->45.91.116.8:4048: use of closed network connection Dec 18 14:26:48.540840511 (310h57m48.411923556s) [noise=true , address=45.91.116.8:4048 , piece_id=BHQYLAJFKTBQ2WGY4XGRI4UOPTGB3SFEUVYZ7XWAC2BKA3ISU3WA, seg_pos={0 0}, upload=272678]: Long tail cancellation due to optimal threshold Dec 18 14:26:48.540888052 (310h57m48.411971107s) [noise=true , address=45.91.116.5:4299 , piece_id=SIZBVTWHVPXBL7OMRRQ467KFLRB56JNA5WDRNSXSNHPSV7UYG5LQ, seg_pos={0 0}, upload=272678]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Dec 18 14:26:48.540899453 (310h57m48.411982498s) [noise=true , address=45.91.116.5:4299 , piece_id=SIZBVTWHVPXBL7OMRRQ467KFLRB56JNA5WDRNSXSNHPSV7UYG5LQ, seg_pos={0 0}, upload=272678]: Long tail cancellation due to optimal threshold Dec 18 14:26:48.541046764 (310h57m48.412129809s) [noise=true , address=77.90.31.11:4556 , piece_id=4GEBIFAQJYJX4KTPHTYUODQZ6ABTTPOA5SYOJBGUV7YDFIHHVO2Q, seg_pos={0 0}, upload=272678]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: expected piece hash; context canceled; EOF Dec 18 14:26:48.541055401 (310h57m48.412138436s) [noise=true , address=77.90.31.11:4556 , piece_id=4GEBIFAQJYJX4KTPHTYUODQZ6ABTTPOA5SYOJBGUV7YDFIHHVO2Q, seg_pos={0 0}, upload=272678]: Long tail cancellation due to optimal threshold Dec 18 14:26:48.541086550 (310h57m48.412169595s) [seg_pos={0 0}, upload=272678]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 18 14:26:48.541155190 (310h57m48.412238226s) [upload=272678]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 18 14:26:48.541159900 (310h57m48.412242935s) [upload=272678]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 18 14:26:48.836195919 (310h57m48.707278954s) [upload=272678]: Done uploading segments... Dec 18 14:26:48.836215697 (310h57m48.707298742s) [upload=272678]: Upload finished. err: <nil> Duration: 5.957770757s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 20 out of 60 is failed (threshold=6) [........X.......X.....X.X.XX...XXXXX.XX..XX.XXXX.....X......] duration: 3.581996686s trace: https://jaeger.dev.storj.io/trace/142285b66b9055cd FAIL (2025-12-18 14:44:30 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.582043345s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..................X....................X....................] duration: 5.781908048s trace: https://jaeger.dev.storj.io/trace/2cbc07c217c30b3f FAIL (2025-12-18 14:43:38 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, Piece ID: O7IY6V6T4UILG3SXQ6K3SPXBVA2VLB22PWVQYIYORGLLYN6LE64A) context deadline exceeded; piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: I2SLWBEKFHZXWYTBZF73DICAZ4ZBXQYU6DES2QWLJMUCFZOCC7CA) context deadline exceeded ======LOGS====== ec 18 14:26:45.953557484 (310h57m45.824640529s) [upload=272678]: Starting upload Dec 18 14:26:45.953631205 (310h57m45.82471425s) [upload=272678]: Done starting upload Dec 18 14:26:45.953727147 (310h57m45.824810202s) [upload=272678]: Uploading segments... Dec 18 14:26:45.953920536 (310h57m45.825003581s) [upload=272678]: Got next segment. Inline: false Dec 18 14:26:45.953955743 (310h57m45.825038798s) [upload=272678]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 18 14:26:45.953963277 (310h57m45.825046322s) [upload=272678]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 18 14:26:46.201381593 (310h57m46.072464628s) [seg_pos={0 0}, upload=272678]: Begin upload segment... Dec 18 14:26:46.201430356 (310h57m46.072513391s) [seg_pos={0 0}, upload=272678]: Done begin upload segment. Dec 18 14:26:46.201523023 (310h57m46.072606088s) [noise=true , address=145.79.227.45:4231 , piece_id=V7B6RPI74SKRJQZCUWFHB2ECD46T3V224VKBL47NIIK5YH6PJCUQ, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.201534584 (310h57m46.072617619s) [noise=true , address=45.91.116.8:4048 , piece_id=BHQYLAJFKTBQ2WGY4XGRI4UOPTGB3SFEUVYZ7XWAC2BKA3ISU3WA, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.203237186 (310h57m46.074320231s) [noise=true , address=45.91.116.14:4190 , piece_id=I2SLWBEKFHZXWYTBZF73DICAZ4ZBXQYU6DES2QWLJMUCFZOCC7CA, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.203669961 (310h57m46.074752996s) [noise=true , address=145.79.227.36:4519 , piece_id=3E54JIZJCR55H36BEOLTDKWRMYA3WHNDPBSALSMNRY45O3BYO6JA, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.203970172 (310h57m46.075053207s) [noise=true , address=145.79.227.38:4978 , piece_id=FRWJ6Z3EH45YRMQVU235FJB75JWK7HOCLZXS7ASITOHLTL3YY5PQ, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.204250205 (310h57m46.07533324s) [noise=true , address=45.91.116.10:4084 , piece_id=UOTQVSUYETMA7UOEURUAT5RPX4QUO4FOXSLQY7UDUKJBCYCOH5BA, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.204534577 (310h57m46.075617612s) [noise=true , address=45.91.116.5:4299 , piece_id=SIZBVTWHVPXBL7OMRRQ467KFLRB56JNA5WDRNSXSNHPSV7UYG5LQ, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.204821995 (310h57m46.07590504s) [noise=true , address=145.79.227.29:4943 , piece_id=4DWWCYWOAU7NSESXDCONC6THXO3TTUYJMN7G6PTG2CP2OOOY2USA, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.206113864 (310h57m46.0771969s) [noise=true , address=77.90.31.11:4556 , piece_id=4GEBIFAQJYJX4KTPHTYUODQZ6ABTTPOA5SYOJBGUV7YDFIHHVO2Q, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.206151877 (310h57m46.077234912s) [noise=true , address=145.79.227.3:4150 , piece_id=XR5E35XW6KBFTREM2GFHARKAAIAINOWMLVYDKKBPFQBXJZOFCC2Q, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.207752204 (310h57m46.078835239s) [noise=true , address=145.79.227.39:4013 , piece_id=2SYWMQU5R7MCTH2E4X732AOVIRVPGUNNXLIN3UFCU7WF4BEJUACQ, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.210366552 (310h57m46.081449597s) [noise=true , address=145.79.227.35:4482 , piece_id=3EGA2CLZBLJWSSULQCBLEEHCMCJDE2FSOY4FCANAQXPEYTWIF53A, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.211613657 (310h57m46.082696692s) [noise=true , address=145.79.227.47:4266 , piece_id=4444BNJTIBUQWLNGTEK3IFD4TIWTCIBCWUHSYBMD3FGZSNDPWT3Q, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.212828561 (310h57m46.083911596s) [noise=true , address=145.79.227.41:4086 , piece_id=DFSNBYWUJ6Z65VTU5ME2IGEDB6KLMTAJM65RC44FUVYHTPEW4ZSA, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.215707031 (310h57m46.086790066s) [noise=true , address=45.91.116.13:4155 , piece_id=O7IY6V6T4UILG3SXQ6K3SPXBVA2VLB22PWVQYIYORGLLYN6LE64A, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.216889753 (310h57m46.087972798s) [noise=true , address=145.79.227.31:4797 , piece_id=F5WHLSRZSSZ4CRJRR2ZPYGGG67SHQ3QMAIHIC42NOVDW2UXARINQ, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:47.912125819 (310h57m47.783208865s) [upload=272678]: Next returned nil segment Dec 18 14:26:47.912143884 (310h57m47.783226929s) [upload=272678]: Waiting for error group managing segments... Dec 18 14:26:48.193212096 (310h57m48.064295131s) [noise=true , address=145.79.227.45:4231 , piece_id=V7B6RPI74SKRJQZCUWFHB2ECD46T3V224VKBL47NIIK5YH6PJCUQ, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.207387668 (310h57m48.078470703s) [noise=true , address=145.79.227.31:4797 , piece_id=F5WHLSRZSSZ4CRJRR2ZPYGGG67SHQ3QMAIHIC42NOVDW2UXARINQ, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.219064714 (310h57m48.090147759s) [noise=true , address=145.79.227.39:4013 , piece_id=2SYWMQU5R7MCTH2E4X732AOVIRVPGUNNXLIN3UFCU7WF4BEJUACQ, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.227887805 (310h57m48.09897085s) [noise=true , address=145.79.227.36:4519 , piece_id=3E54JIZJCR55H36BEOLTDKWRMYA3WHNDPBSALSMNRY45O3BYO6JA, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.229044407 (310h57m48.100127442s) [noise=true , address=145.79.227.41:4086 , piece_id=DFSNBYWUJ6Z65VTU5ME2IGEDB6KLMTAJM65RC44FUVYHTPEW4ZSA, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.234560812 (310h57m48.105643857s) [noise=true , address=145.79.227.47:4266 , piece_id=4444BNJTIBUQWLNGTEK3IFD4TIWTCIBCWUHSYBMD3FGZSNDPWT3Q, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.237048037 (310h57m48.108131072s) [noise=true , address=145.79.227.35:4482 , piece_id=3EGA2CLZBLJWSSULQCBLEEHCMCJDE2FSOY4FCANAQXPEYTWIF53A, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.267241984 (310h57m48.138325019s) [noise=true , address=145.79.227.29:4943 , piece_id=4DWWCYWOAU7NSESXDCONC6THXO3TTUYJMN7G6PTG2CP2OOOY2USA, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.288475576 (310h57m48.159558621s) [noise=true , address=145.79.227.3:4150 , piece_id=XR5E35XW6KBFTREM2GFHARKAAIAINOWMLVYDKKBPFQBXJZOFCC2Q, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.293478393 (310h57m48.164561458s) [noise=true , address=145.79.227.38:4978 , piece_id=FRWJ6Z3EH45YRMQVU235FJB75JWK7HOCLZXS7ASITOHLTL3YY5PQ, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.423759785 (310h57m48.29484282s) [noise=true , address=45.91.116.14:4190 , piece_id=I2SLWBEKFHZXWYTBZF73DICAZ4ZBXQYU6DES2QWLJMUCFZOCC7CA, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.485697347 (310h57m48.356780392s) [noise=true , address=45.91.116.10:4084 , piece_id=UOTQVSUYETMA7UOEURUAT5RPX4QUO4FOXSLQY7UDUKJBCYCOH5BA, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.540540711 (310h57m48.411623756s) [noise=true , address=45.91.116.13:4155 , piece_id=O7IY6V6T4UILG3SXQ6K3SPXBVA2VLB22PWVQYIYORGLLYN6LE64A, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.540575787 (310h57m48.411658832s) [seg_pos={0 0}, upload=272678]: Segment reached optimal threshold of 13 pieces. Dec 18 14:26:48.540831424 (310h57m48.411914459s) [noise=true , address=45.91.116.8:4048 , piece_id=BHQYLAJFKTBQ2WGY4XGRI4UOPTGB3SFEUVYZ7XWAC2BKA3ISU3WA, seg_pos={0 0}, upload=272678]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: write tcp 188.166.225.170:49328->45.91.116.8:4048: use of closed network connection; write tcp 188.166.225.170:49328->45.91.116.8:4048: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49328->45.91.116.8:4048: use of closed network connection Dec 18 14:26:48.540840511 (310h57m48.411923556s) [noise=true , address=45.91.116.8:4048 , piece_id=BHQYLAJFKTBQ2WGY4XGRI4UOPTGB3SFEUVYZ7XWAC2BKA3ISU3WA, seg_pos={0 0}, upload=272678]: Long tail cancellation due to optimal threshold Dec 18 14:26:48.540888052 (310h57m48.411971107s) [noise=true , address=45.91.116.5:4299 , piece_id=SIZBVTWHVPXBL7OMRRQ467KFLRB56JNA5WDRNSXSNHPSV7UYG5LQ, seg_pos={0 0}, upload=272678]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Dec 18 14:26:48.540899453 (310h57m48.411982498s) [noise=true , address=45.91.116.5:4299 , piece_id=SIZBVTWHVPXBL7OMRRQ467KFLRB56JNA5WDRNSXSNHPSV7UYG5LQ, seg_pos={0 0}, upload=272678]: Long tail cancellation due to optimal threshold Dec 18 14:26:48.541046764 (310h57m48.412129809s) [noise=true , address=77.90.31.11:4556 , piece_id=4GEBIFAQJYJX4KTPHTYUODQZ6ABTTPOA5SYOJBGUV7YDFIHHVO2Q, seg_pos={0 0}, upload=272678]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: expected piece hash; context canceled; EOF Dec 18 14:26:48.541055401 (310h57m48.412138436s) [noise=true , address=77.90.31.11:4556 , piece_id=4GEBIFAQJYJX4KTPHTYUODQZ6ABTTPOA5SYOJBGUV7YDFIHHVO2Q, seg_pos={0 0}, upload=272678]: Long tail cancellation due to optimal threshold Dec 18 14:26:48.541086550 (310h57m48.412169595s) [seg_pos={0 0}, upload=272678]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 18 14:26:48.541155190 (310h57m48.412238226s) [upload=272678]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 18 14:26:48.541159900 (310h57m48.412242935s) [upload=272678]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 18 14:26:48.836195919 (310h57m48.707278954s) [upload=272678]: Done uploading segments... Dec 18 14:26:48.836215697 (310h57m48.707298742s) [upload=272678]: Upload finished. err: <nil> Duration: 5.781937494s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 20 out of 60 is failed (threshold=6) [.........X.......X.....X.X.XX...XXXXX.XX..XX.XXXX.....X.....] duration: 4.662126033s trace: https://jaeger.dev.storj.io/trace/142285b66b9055cd FAIL (2025-12-18 14:43:31 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.662153875s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...................X....................X...................] duration: 6.021836412s trace: https://jaeger.dev.storj.io/trace/2cbc07c217c30b3f FAIL (2025-12-18 14:42:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, Piece ID: O7IY6V6T4UILG3SXQ6K3SPXBVA2VLB22PWVQYIYORGLLYN6LE64A) context deadline exceeded; piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: I2SLWBEKFHZXWYTBZF73DICAZ4ZBXQYU6DES2QWLJMUCFZOCC7CA) context deadline exceeded ======LOGS====== ec 18 14:26:45.953557484 (310h57m45.824640529s) [upload=272678]: Starting upload Dec 18 14:26:45.953631205 (310h57m45.82471425s) [upload=272678]: Done starting upload Dec 18 14:26:45.953727147 (310h57m45.824810202s) [upload=272678]: Uploading segments... Dec 18 14:26:45.953920536 (310h57m45.825003581s) [upload=272678]: Got next segment. Inline: false Dec 18 14:26:45.953955743 (310h57m45.825038798s) [upload=272678]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 18 14:26:45.953963277 (310h57m45.825046322s) [upload=272678]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 18 14:26:46.201381593 (310h57m46.072464628s) [seg_pos={0 0}, upload=272678]: Begin upload segment... Dec 18 14:26:46.201430356 (310h57m46.072513391s) [seg_pos={0 0}, upload=272678]: Done begin upload segment. Dec 18 14:26:46.201523023 (310h57m46.072606088s) [noise=true , address=145.79.227.45:4231 , piece_id=V7B6RPI74SKRJQZCUWFHB2ECD46T3V224VKBL47NIIK5YH6PJCUQ, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.201534584 (310h57m46.072617619s) [noise=true , address=45.91.116.8:4048 , piece_id=BHQYLAJFKTBQ2WGY4XGRI4UOPTGB3SFEUVYZ7XWAC2BKA3ISU3WA, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.203237186 (310h57m46.074320231s) [noise=true , address=45.91.116.14:4190 , piece_id=I2SLWBEKFHZXWYTBZF73DICAZ4ZBXQYU6DES2QWLJMUCFZOCC7CA, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.203669961 (310h57m46.074752996s) [noise=true , address=145.79.227.36:4519 , piece_id=3E54JIZJCR55H36BEOLTDKWRMYA3WHNDPBSALSMNRY45O3BYO6JA, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.203970172 (310h57m46.075053207s) [noise=true , address=145.79.227.38:4978 , piece_id=FRWJ6Z3EH45YRMQVU235FJB75JWK7HOCLZXS7ASITOHLTL3YY5PQ, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.204250205 (310h57m46.07533324s) [noise=true , address=45.91.116.10:4084 , piece_id=UOTQVSUYETMA7UOEURUAT5RPX4QUO4FOXSLQY7UDUKJBCYCOH5BA, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.204534577 (310h57m46.075617612s) [noise=true , address=45.91.116.5:4299 , piece_id=SIZBVTWHVPXBL7OMRRQ467KFLRB56JNA5WDRNSXSNHPSV7UYG5LQ, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.204821995 (310h57m46.07590504s) [noise=true , address=145.79.227.29:4943 , piece_id=4DWWCYWOAU7NSESXDCONC6THXO3TTUYJMN7G6PTG2CP2OOOY2USA, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.206113864 (310h57m46.0771969s) [noise=true , address=77.90.31.11:4556 , piece_id=4GEBIFAQJYJX4KTPHTYUODQZ6ABTTPOA5SYOJBGUV7YDFIHHVO2Q, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.206151877 (310h57m46.077234912s) [noise=true , address=145.79.227.3:4150 , piece_id=XR5E35XW6KBFTREM2GFHARKAAIAINOWMLVYDKKBPFQBXJZOFCC2Q, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.207752204 (310h57m46.078835239s) [noise=true , address=145.79.227.39:4013 , piece_id=2SYWMQU5R7MCTH2E4X732AOVIRVPGUNNXLIN3UFCU7WF4BEJUACQ, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.210366552 (310h57m46.081449597s) [noise=true , address=145.79.227.35:4482 , piece_id=3EGA2CLZBLJWSSULQCBLEEHCMCJDE2FSOY4FCANAQXPEYTWIF53A, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.211613657 (310h57m46.082696692s) [noise=true , address=145.79.227.47:4266 , piece_id=4444BNJTIBUQWLNGTEK3IFD4TIWTCIBCWUHSYBMD3FGZSNDPWT3Q, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.212828561 (310h57m46.083911596s) [noise=true , address=145.79.227.41:4086 , piece_id=DFSNBYWUJ6Z65VTU5ME2IGEDB6KLMTAJM65RC44FUVYHTPEW4ZSA, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.215707031 (310h57m46.086790066s) [noise=true , address=45.91.116.13:4155 , piece_id=O7IY6V6T4UILG3SXQ6K3SPXBVA2VLB22PWVQYIYORGLLYN6LE64A, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.216889753 (310h57m46.087972798s) [noise=true , address=145.79.227.31:4797 , piece_id=F5WHLSRZSSZ4CRJRR2ZPYGGG67SHQ3QMAIHIC42NOVDW2UXARINQ, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:47.912125819 (310h57m47.783208865s) [upload=272678]: Next returned nil segment Dec 18 14:26:47.912143884 (310h57m47.783226929s) [upload=272678]: Waiting for error group managing segments... Dec 18 14:26:48.193212096 (310h57m48.064295131s) [noise=true , address=145.79.227.45:4231 , piece_id=V7B6RPI74SKRJQZCUWFHB2ECD46T3V224VKBL47NIIK5YH6PJCUQ, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.207387668 (310h57m48.078470703s) [noise=true , address=145.79.227.31:4797 , piece_id=F5WHLSRZSSZ4CRJRR2ZPYGGG67SHQ3QMAIHIC42NOVDW2UXARINQ, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.219064714 (310h57m48.090147759s) [noise=true , address=145.79.227.39:4013 , piece_id=2SYWMQU5R7MCTH2E4X732AOVIRVPGUNNXLIN3UFCU7WF4BEJUACQ, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.227887805 (310h57m48.09897085s) [noise=true , address=145.79.227.36:4519 , piece_id=3E54JIZJCR55H36BEOLTDKWRMYA3WHNDPBSALSMNRY45O3BYO6JA, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.229044407 (310h57m48.100127442s) [noise=true , address=145.79.227.41:4086 , piece_id=DFSNBYWUJ6Z65VTU5ME2IGEDB6KLMTAJM65RC44FUVYHTPEW4ZSA, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.234560812 (310h57m48.105643857s) [noise=true , address=145.79.227.47:4266 , piece_id=4444BNJTIBUQWLNGTEK3IFD4TIWTCIBCWUHSYBMD3FGZSNDPWT3Q, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.237048037 (310h57m48.108131072s) [noise=true , address=145.79.227.35:4482 , piece_id=3EGA2CLZBLJWSSULQCBLEEHCMCJDE2FSOY4FCANAQXPEYTWIF53A, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.267241984 (310h57m48.138325019s) [noise=true , address=145.79.227.29:4943 , piece_id=4DWWCYWOAU7NSESXDCONC6THXO3TTUYJMN7G6PTG2CP2OOOY2USA, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.288475576 (310h57m48.159558621s) [noise=true , address=145.79.227.3:4150 , piece_id=XR5E35XW6KBFTREM2GFHARKAAIAINOWMLVYDKKBPFQBXJZOFCC2Q, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.293478393 (310h57m48.164561458s) [noise=true , address=145.79.227.38:4978 , piece_id=FRWJ6Z3EH45YRMQVU235FJB75JWK7HOCLZXS7ASITOHLTL3YY5PQ, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.423759785 (310h57m48.29484282s) [noise=true , address=45.91.116.14:4190 , piece_id=I2SLWBEKFHZXWYTBZF73DICAZ4ZBXQYU6DES2QWLJMUCFZOCC7CA, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.485697347 (310h57m48.356780392s) [noise=true , address=45.91.116.10:4084 , piece_id=UOTQVSUYETMA7UOEURUAT5RPX4QUO4FOXSLQY7UDUKJBCYCOH5BA, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.540540711 (310h57m48.411623756s) [noise=true , address=45.91.116.13:4155 , piece_id=O7IY6V6T4UILG3SXQ6K3SPXBVA2VLB22PWVQYIYORGLLYN6LE64A, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.540575787 (310h57m48.411658832s) [seg_pos={0 0}, upload=272678]: Segment reached optimal threshold of 13 pieces. Dec 18 14:26:48.540831424 (310h57m48.411914459s) [noise=true , address=45.91.116.8:4048 , piece_id=BHQYLAJFKTBQ2WGY4XGRI4UOPTGB3SFEUVYZ7XWAC2BKA3ISU3WA, seg_pos={0 0}, upload=272678]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: write tcp 188.166.225.170:49328->45.91.116.8:4048: use of closed network connection; write tcp 188.166.225.170:49328->45.91.116.8:4048: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49328->45.91.116.8:4048: use of closed network connection Dec 18 14:26:48.540840511 (310h57m48.411923556s) [noise=true , address=45.91.116.8:4048 , piece_id=BHQYLAJFKTBQ2WGY4XGRI4UOPTGB3SFEUVYZ7XWAC2BKA3ISU3WA, seg_pos={0 0}, upload=272678]: Long tail cancellation due to optimal threshold Dec 18 14:26:48.540888052 (310h57m48.411971107s) [noise=true , address=45.91.116.5:4299 , piece_id=SIZBVTWHVPXBL7OMRRQ467KFLRB56JNA5WDRNSXSNHPSV7UYG5LQ, seg_pos={0 0}, upload=272678]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Dec 18 14:26:48.540899453 (310h57m48.411982498s) [noise=true , address=45.91.116.5:4299 , piece_id=SIZBVTWHVPXBL7OMRRQ467KFLRB56JNA5WDRNSXSNHPSV7UYG5LQ, seg_pos={0 0}, upload=272678]: Long tail cancellation due to optimal threshold Dec 18 14:26:48.541046764 (310h57m48.412129809s) [noise=true , address=77.90.31.11:4556 , piece_id=4GEBIFAQJYJX4KTPHTYUODQZ6ABTTPOA5SYOJBGUV7YDFIHHVO2Q, seg_pos={0 0}, upload=272678]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: expected piece hash; context canceled; EOF Dec 18 14:26:48.541055401 (310h57m48.412138436s) [noise=true , address=77.90.31.11:4556 , piece_id=4GEBIFAQJYJX4KTPHTYUODQZ6ABTTPOA5SYOJBGUV7YDFIHHVO2Q, seg_pos={0 0}, upload=272678]: Long tail cancellation due to optimal threshold Dec 18 14:26:48.541086550 (310h57m48.412169595s) [seg_pos={0 0}, upload=272678]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 18 14:26:48.541155190 (310h57m48.412238226s) [upload=272678]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 18 14:26:48.541159900 (310h57m48.412242935s) [upload=272678]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 18 14:26:48.836195919 (310h57m48.707278954s) [upload=272678]: Done uploading segments... Dec 18 14:26:48.836215697 (310h57m48.707298742s) [upload=272678]: Upload finished. err: <nil> Duration: 6.021860568s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 20 out of 60 is failed (threshold=6) [..........X.......X.....X.X.XX...XXXXX.XX..XX.XXXX.....X....] duration: 5.001106518s trace: https://jaeger.dev.storj.io/trace/142285b66b9055cd FAIL (2025-12-18 14:42:19 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001128209s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [....................X....................X..................] duration: 5.997789327s trace: https://jaeger.dev.storj.io/trace/2cbc07c217c30b3f FAIL (2025-12-18 14:41:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, Piece ID: O7IY6V6T4UILG3SXQ6K3SPXBVA2VLB22PWVQYIYORGLLYN6LE64A) context deadline exceeded; piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: I2SLWBEKFHZXWYTBZF73DICAZ4ZBXQYU6DES2QWLJMUCFZOCC7CA) context deadline exceeded ======LOGS====== ec 18 14:26:45.953557484 (310h57m45.824640529s) [upload=272678]: Starting upload Dec 18 14:26:45.953631205 (310h57m45.82471425s) [upload=272678]: Done starting upload Dec 18 14:26:45.953727147 (310h57m45.824810202s) [upload=272678]: Uploading segments... Dec 18 14:26:45.953920536 (310h57m45.825003581s) [upload=272678]: Got next segment. Inline: false Dec 18 14:26:45.953955743 (310h57m45.825038798s) [upload=272678]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 18 14:26:45.953963277 (310h57m45.825046322s) [upload=272678]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 18 14:26:46.201381593 (310h57m46.072464628s) [seg_pos={0 0}, upload=272678]: Begin upload segment... Dec 18 14:26:46.201430356 (310h57m46.072513391s) [seg_pos={0 0}, upload=272678]: Done begin upload segment. Dec 18 14:26:46.201523023 (310h57m46.072606088s) [noise=true , address=145.79.227.45:4231 , piece_id=V7B6RPI74SKRJQZCUWFHB2ECD46T3V224VKBL47NIIK5YH6PJCUQ, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.201534584 (310h57m46.072617619s) [noise=true , address=45.91.116.8:4048 , piece_id=BHQYLAJFKTBQ2WGY4XGRI4UOPTGB3SFEUVYZ7XWAC2BKA3ISU3WA, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.203237186 (310h57m46.074320231s) [noise=true , address=45.91.116.14:4190 , piece_id=I2SLWBEKFHZXWYTBZF73DICAZ4ZBXQYU6DES2QWLJMUCFZOCC7CA, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.203669961 (310h57m46.074752996s) [noise=true , address=145.79.227.36:4519 , piece_id=3E54JIZJCR55H36BEOLTDKWRMYA3WHNDPBSALSMNRY45O3BYO6JA, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.203970172 (310h57m46.075053207s) [noise=true , address=145.79.227.38:4978 , piece_id=FRWJ6Z3EH45YRMQVU235FJB75JWK7HOCLZXS7ASITOHLTL3YY5PQ, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.204250205 (310h57m46.07533324s) [noise=true , address=45.91.116.10:4084 , piece_id=UOTQVSUYETMA7UOEURUAT5RPX4QUO4FOXSLQY7UDUKJBCYCOH5BA, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.204534577 (310h57m46.075617612s) [noise=true , address=45.91.116.5:4299 , piece_id=SIZBVTWHVPXBL7OMRRQ467KFLRB56JNA5WDRNSXSNHPSV7UYG5LQ, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.204821995 (310h57m46.07590504s) [noise=true , address=145.79.227.29:4943 , piece_id=4DWWCYWOAU7NSESXDCONC6THXO3TTUYJMN7G6PTG2CP2OOOY2USA, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.206113864 (310h57m46.0771969s) [noise=true , address=77.90.31.11:4556 , piece_id=4GEBIFAQJYJX4KTPHTYUODQZ6ABTTPOA5SYOJBGUV7YDFIHHVO2Q, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.206151877 (310h57m46.077234912s) [noise=true , address=145.79.227.3:4150 , piece_id=XR5E35XW6KBFTREM2GFHARKAAIAINOWMLVYDKKBPFQBXJZOFCC2Q, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.207752204 (310h57m46.078835239s) [noise=true , address=145.79.227.39:4013 , piece_id=2SYWMQU5R7MCTH2E4X732AOVIRVPGUNNXLIN3UFCU7WF4BEJUACQ, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.210366552 (310h57m46.081449597s) [noise=true , address=145.79.227.35:4482 , piece_id=3EGA2CLZBLJWSSULQCBLEEHCMCJDE2FSOY4FCANAQXPEYTWIF53A, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.211613657 (310h57m46.082696692s) [noise=true , address=145.79.227.47:4266 , piece_id=4444BNJTIBUQWLNGTEK3IFD4TIWTCIBCWUHSYBMD3FGZSNDPWT3Q, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.212828561 (310h57m46.083911596s) [noise=true , address=145.79.227.41:4086 , piece_id=DFSNBYWUJ6Z65VTU5ME2IGEDB6KLMTAJM65RC44FUVYHTPEW4ZSA, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.215707031 (310h57m46.086790066s) [noise=true , address=45.91.116.13:4155 , piece_id=O7IY6V6T4UILG3SXQ6K3SPXBVA2VLB22PWVQYIYORGLLYN6LE64A, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.216889753 (310h57m46.087972798s) [noise=true , address=145.79.227.31:4797 , piece_id=F5WHLSRZSSZ4CRJRR2ZPYGGG67SHQ3QMAIHIC42NOVDW2UXARINQ, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:47.912125819 (310h57m47.783208865s) [upload=272678]: Next returned nil segment Dec 18 14:26:47.912143884 (310h57m47.783226929s) [upload=272678]: Waiting for error group managing segments... Dec 18 14:26:48.193212096 (310h57m48.064295131s) [noise=true , address=145.79.227.45:4231 , piece_id=V7B6RPI74SKRJQZCUWFHB2ECD46T3V224VKBL47NIIK5YH6PJCUQ, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.207387668 (310h57m48.078470703s) [noise=true , address=145.79.227.31:4797 , piece_id=F5WHLSRZSSZ4CRJRR2ZPYGGG67SHQ3QMAIHIC42NOVDW2UXARINQ, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.219064714 (310h57m48.090147759s) [noise=true , address=145.79.227.39:4013 , piece_id=2SYWMQU5R7MCTH2E4X732AOVIRVPGUNNXLIN3UFCU7WF4BEJUACQ, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.227887805 (310h57m48.09897085s) [noise=true , address=145.79.227.36:4519 , piece_id=3E54JIZJCR55H36BEOLTDKWRMYA3WHNDPBSALSMNRY45O3BYO6JA, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.229044407 (310h57m48.100127442s) [noise=true , address=145.79.227.41:4086 , piece_id=DFSNBYWUJ6Z65VTU5ME2IGEDB6KLMTAJM65RC44FUVYHTPEW4ZSA, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.234560812 (310h57m48.105643857s) [noise=true , address=145.79.227.47:4266 , piece_id=4444BNJTIBUQWLNGTEK3IFD4TIWTCIBCWUHSYBMD3FGZSNDPWT3Q, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.237048037 (310h57m48.108131072s) [noise=true , address=145.79.227.35:4482 , piece_id=3EGA2CLZBLJWSSULQCBLEEHCMCJDE2FSOY4FCANAQXPEYTWIF53A, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.267241984 (310h57m48.138325019s) [noise=true , address=145.79.227.29:4943 , piece_id=4DWWCYWOAU7NSESXDCONC6THXO3TTUYJMN7G6PTG2CP2OOOY2USA, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.288475576 (310h57m48.159558621s) [noise=true , address=145.79.227.3:4150 , piece_id=XR5E35XW6KBFTREM2GFHARKAAIAINOWMLVYDKKBPFQBXJZOFCC2Q, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.293478393 (310h57m48.164561458s) [noise=true , address=145.79.227.38:4978 , piece_id=FRWJ6Z3EH45YRMQVU235FJB75JWK7HOCLZXS7ASITOHLTL3YY5PQ, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.423759785 (310h57m48.29484282s) [noise=true , address=45.91.116.14:4190 , piece_id=I2SLWBEKFHZXWYTBZF73DICAZ4ZBXQYU6DES2QWLJMUCFZOCC7CA, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.485697347 (310h57m48.356780392s) [noise=true , address=45.91.116.10:4084 , piece_id=UOTQVSUYETMA7UOEURUAT5RPX4QUO4FOXSLQY7UDUKJBCYCOH5BA, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.540540711 (310h57m48.411623756s) [noise=true , address=45.91.116.13:4155 , piece_id=O7IY6V6T4UILG3SXQ6K3SPXBVA2VLB22PWVQYIYORGLLYN6LE64A, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.540575787 (310h57m48.411658832s) [seg_pos={0 0}, upload=272678]: Segment reached optimal threshold of 13 pieces. Dec 18 14:26:48.540831424 (310h57m48.411914459s) [noise=true , address=45.91.116.8:4048 , piece_id=BHQYLAJFKTBQ2WGY4XGRI4UOPTGB3SFEUVYZ7XWAC2BKA3ISU3WA, seg_pos={0 0}, upload=272678]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: write tcp 188.166.225.170:49328->45.91.116.8:4048: use of closed network connection; write tcp 188.166.225.170:49328->45.91.116.8:4048: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49328->45.91.116.8:4048: use of closed network connection Dec 18 14:26:48.540840511 (310h57m48.411923556s) [noise=true , address=45.91.116.8:4048 , piece_id=BHQYLAJFKTBQ2WGY4XGRI4UOPTGB3SFEUVYZ7XWAC2BKA3ISU3WA, seg_pos={0 0}, upload=272678]: Long tail cancellation due to optimal threshold Dec 18 14:26:48.540888052 (310h57m48.411971107s) [noise=true , address=45.91.116.5:4299 , piece_id=SIZBVTWHVPXBL7OMRRQ467KFLRB56JNA5WDRNSXSNHPSV7UYG5LQ, seg_pos={0 0}, upload=272678]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Dec 18 14:26:48.540899453 (310h57m48.411982498s) [noise=true , address=45.91.116.5:4299 , piece_id=SIZBVTWHVPXBL7OMRRQ467KFLRB56JNA5WDRNSXSNHPSV7UYG5LQ, seg_pos={0 0}, upload=272678]: Long tail cancellation due to optimal threshold Dec 18 14:26:48.541046764 (310h57m48.412129809s) [noise=true , address=77.90.31.11:4556 , piece_id=4GEBIFAQJYJX4KTPHTYUODQZ6ABTTPOA5SYOJBGUV7YDFIHHVO2Q, seg_pos={0 0}, upload=272678]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: expected piece hash; context canceled; EOF Dec 18 14:26:48.541055401 (310h57m48.412138436s) [noise=true , address=77.90.31.11:4556 , piece_id=4GEBIFAQJYJX4KTPHTYUODQZ6ABTTPOA5SYOJBGUV7YDFIHHVO2Q, seg_pos={0 0}, upload=272678]: Long tail cancellation due to optimal threshold Dec 18 14:26:48.541086550 (310h57m48.412169595s) [seg_pos={0 0}, upload=272678]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 18 14:26:48.541155190 (310h57m48.412238226s) [upload=272678]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 18 14:26:48.541159900 (310h57m48.412242935s) [upload=272678]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 18 14:26:48.836195919 (310h57m48.707278954s) [upload=272678]: Done uploading segments... Dec 18 14:26:48.836215697 (310h57m48.707298742s) [upload=272678]: Upload finished. err: <nil> Duration: 5.997808192s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 20 out of 60 is failed (threshold=6) [...........X.......X.....X.X.XX...XXXXX.XX..XX.XXXX.....X...] duration: 4.657481068s trace: https://jaeger.dev.storj.io/trace/142285b66b9055cd FAIL (2025-12-18 14:41:31 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.657506065s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 20 out of 60 is failed (threshold=6) [............X.......X.....X.X.XX...XXXXX.XX..XX.XXXX.....X..] duration: 3.405565545s trace: https://jaeger.dev.storj.io/trace/142285b66b9055cd FAIL (2025-12-18 14:41:14 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.405593877s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 20 out of 60 is failed (threshold=6) [.............X.......X.....X.X.XX...XXXXX.XX..XX.XXXX.....X.] duration: 5.000541314s trace: https://jaeger.dev.storj.io/trace/142285b66b9055cd FAIL (2025-12-18 14:40:47 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000566703s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....................X....................X.................] duration: 6.206428642s trace: https://jaeger.dev.storj.io/trace/2cbc07c217c30b3f FAIL (2025-12-18 14:40:38 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, Piece ID: O7IY6V6T4UILG3SXQ6K3SPXBVA2VLB22PWVQYIYORGLLYN6LE64A) context deadline exceeded; piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: I2SLWBEKFHZXWYTBZF73DICAZ4ZBXQYU6DES2QWLJMUCFZOCC7CA) context deadline exceeded ======LOGS====== ec 18 14:26:45.953557484 (310h57m45.824640529s) [upload=272678]: Starting upload Dec 18 14:26:45.953631205 (310h57m45.82471425s) [upload=272678]: Done starting upload Dec 18 14:26:45.953727147 (310h57m45.824810202s) [upload=272678]: Uploading segments... Dec 18 14:26:45.953920536 (310h57m45.825003581s) [upload=272678]: Got next segment. Inline: false Dec 18 14:26:45.953955743 (310h57m45.825038798s) [upload=272678]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 18 14:26:45.953963277 (310h57m45.825046322s) [upload=272678]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 18 14:26:46.201381593 (310h57m46.072464628s) [seg_pos={0 0}, upload=272678]: Begin upload segment... Dec 18 14:26:46.201430356 (310h57m46.072513391s) [seg_pos={0 0}, upload=272678]: Done begin upload segment. Dec 18 14:26:46.201523023 (310h57m46.072606088s) [noise=true , address=145.79.227.45:4231 , piece_id=V7B6RPI74SKRJQZCUWFHB2ECD46T3V224VKBL47NIIK5YH6PJCUQ, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.201534584 (310h57m46.072617619s) [noise=true , address=45.91.116.8:4048 , piece_id=BHQYLAJFKTBQ2WGY4XGRI4UOPTGB3SFEUVYZ7XWAC2BKA3ISU3WA, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.203237186 (310h57m46.074320231s) [noise=true , address=45.91.116.14:4190 , piece_id=I2SLWBEKFHZXWYTBZF73DICAZ4ZBXQYU6DES2QWLJMUCFZOCC7CA, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.203669961 (310h57m46.074752996s) [noise=true , address=145.79.227.36:4519 , piece_id=3E54JIZJCR55H36BEOLTDKWRMYA3WHNDPBSALSMNRY45O3BYO6JA, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.203970172 (310h57m46.075053207s) [noise=true , address=145.79.227.38:4978 , piece_id=FRWJ6Z3EH45YRMQVU235FJB75JWK7HOCLZXS7ASITOHLTL3YY5PQ, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.204250205 (310h57m46.07533324s) [noise=true , address=45.91.116.10:4084 , piece_id=UOTQVSUYETMA7UOEURUAT5RPX4QUO4FOXSLQY7UDUKJBCYCOH5BA, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.204534577 (310h57m46.075617612s) [noise=true , address=45.91.116.5:4299 , piece_id=SIZBVTWHVPXBL7OMRRQ467KFLRB56JNA5WDRNSXSNHPSV7UYG5LQ, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.204821995 (310h57m46.07590504s) [noise=true , address=145.79.227.29:4943 , piece_id=4DWWCYWOAU7NSESXDCONC6THXO3TTUYJMN7G6PTG2CP2OOOY2USA, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.206113864 (310h57m46.0771969s) [noise=true , address=77.90.31.11:4556 , piece_id=4GEBIFAQJYJX4KTPHTYUODQZ6ABTTPOA5SYOJBGUV7YDFIHHVO2Q, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.206151877 (310h57m46.077234912s) [noise=true , address=145.79.227.3:4150 , piece_id=XR5E35XW6KBFTREM2GFHARKAAIAINOWMLVYDKKBPFQBXJZOFCC2Q, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.207752204 (310h57m46.078835239s) [noise=true , address=145.79.227.39:4013 , piece_id=2SYWMQU5R7MCTH2E4X732AOVIRVPGUNNXLIN3UFCU7WF4BEJUACQ, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.210366552 (310h57m46.081449597s) [noise=true , address=145.79.227.35:4482 , piece_id=3EGA2CLZBLJWSSULQCBLEEHCMCJDE2FSOY4FCANAQXPEYTWIF53A, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.211613657 (310h57m46.082696692s) [noise=true , address=145.79.227.47:4266 , piece_id=4444BNJTIBUQWLNGTEK3IFD4TIWTCIBCWUHSYBMD3FGZSNDPWT3Q, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.212828561 (310h57m46.083911596s) [noise=true , address=145.79.227.41:4086 , piece_id=DFSNBYWUJ6Z65VTU5ME2IGEDB6KLMTAJM65RC44FUVYHTPEW4ZSA, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.215707031 (310h57m46.086790066s) [noise=true , address=45.91.116.13:4155 , piece_id=O7IY6V6T4UILG3SXQ6K3SPXBVA2VLB22PWVQYIYORGLLYN6LE64A, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.216889753 (310h57m46.087972798s) [noise=true , address=145.79.227.31:4797 , piece_id=F5WHLSRZSSZ4CRJRR2ZPYGGG67SHQ3QMAIHIC42NOVDW2UXARINQ, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:47.912125819 (310h57m47.783208865s) [upload=272678]: Next returned nil segment Dec 18 14:26:47.912143884 (310h57m47.783226929s) [upload=272678]: Waiting for error group managing segments... Dec 18 14:26:48.193212096 (310h57m48.064295131s) [noise=true , address=145.79.227.45:4231 , piece_id=V7B6RPI74SKRJQZCUWFHB2ECD46T3V224VKBL47NIIK5YH6PJCUQ, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.207387668 (310h57m48.078470703s) [noise=true , address=145.79.227.31:4797 , piece_id=F5WHLSRZSSZ4CRJRR2ZPYGGG67SHQ3QMAIHIC42NOVDW2UXARINQ, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.219064714 (310h57m48.090147759s) [noise=true , address=145.79.227.39:4013 , piece_id=2SYWMQU5R7MCTH2E4X732AOVIRVPGUNNXLIN3UFCU7WF4BEJUACQ, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.227887805 (310h57m48.09897085s) [noise=true , address=145.79.227.36:4519 , piece_id=3E54JIZJCR55H36BEOLTDKWRMYA3WHNDPBSALSMNRY45O3BYO6JA, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.229044407 (310h57m48.100127442s) [noise=true , address=145.79.227.41:4086 , piece_id=DFSNBYWUJ6Z65VTU5ME2IGEDB6KLMTAJM65RC44FUVYHTPEW4ZSA, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.234560812 (310h57m48.105643857s) [noise=true , address=145.79.227.47:4266 , piece_id=4444BNJTIBUQWLNGTEK3IFD4TIWTCIBCWUHSYBMD3FGZSNDPWT3Q, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.237048037 (310h57m48.108131072s) [noise=true , address=145.79.227.35:4482 , piece_id=3EGA2CLZBLJWSSULQCBLEEHCMCJDE2FSOY4FCANAQXPEYTWIF53A, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.267241984 (310h57m48.138325019s) [noise=true , address=145.79.227.29:4943 , piece_id=4DWWCYWOAU7NSESXDCONC6THXO3TTUYJMN7G6PTG2CP2OOOY2USA, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.288475576 (310h57m48.159558621s) [noise=true , address=145.79.227.3:4150 , piece_id=XR5E35XW6KBFTREM2GFHARKAAIAINOWMLVYDKKBPFQBXJZOFCC2Q, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.293478393 (310h57m48.164561458s) [noise=true , address=145.79.227.38:4978 , piece_id=FRWJ6Z3EH45YRMQVU235FJB75JWK7HOCLZXS7ASITOHLTL3YY5PQ, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.423759785 (310h57m48.29484282s) [noise=true , address=45.91.116.14:4190 , piece_id=I2SLWBEKFHZXWYTBZF73DICAZ4ZBXQYU6DES2QWLJMUCFZOCC7CA, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.485697347 (310h57m48.356780392s) [noise=true , address=45.91.116.10:4084 , piece_id=UOTQVSUYETMA7UOEURUAT5RPX4QUO4FOXSLQY7UDUKJBCYCOH5BA, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.540540711 (310h57m48.411623756s) [noise=true , address=45.91.116.13:4155 , piece_id=O7IY6V6T4UILG3SXQ6K3SPXBVA2VLB22PWVQYIYORGLLYN6LE64A, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.540575787 (310h57m48.411658832s) [seg_pos={0 0}, upload=272678]: Segment reached optimal threshold of 13 pieces. Dec 18 14:26:48.540831424 (310h57m48.411914459s) [noise=true , address=45.91.116.8:4048 , piece_id=BHQYLAJFKTBQ2WGY4XGRI4UOPTGB3SFEUVYZ7XWAC2BKA3ISU3WA, seg_pos={0 0}, upload=272678]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: write tcp 188.166.225.170:49328->45.91.116.8:4048: use of closed network connection; write tcp 188.166.225.170:49328->45.91.116.8:4048: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49328->45.91.116.8:4048: use of closed network connection Dec 18 14:26:48.540840511 (310h57m48.411923556s) [noise=true , address=45.91.116.8:4048 , piece_id=BHQYLAJFKTBQ2WGY4XGRI4UOPTGB3SFEUVYZ7XWAC2BKA3ISU3WA, seg_pos={0 0}, upload=272678]: Long tail cancellation due to optimal threshold Dec 18 14:26:48.540888052 (310h57m48.411971107s) [noise=true , address=45.91.116.5:4299 , piece_id=SIZBVTWHVPXBL7OMRRQ467KFLRB56JNA5WDRNSXSNHPSV7UYG5LQ, seg_pos={0 0}, upload=272678]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Dec 18 14:26:48.540899453 (310h57m48.411982498s) [noise=true , address=45.91.116.5:4299 , piece_id=SIZBVTWHVPXBL7OMRRQ467KFLRB56JNA5WDRNSXSNHPSV7UYG5LQ, seg_pos={0 0}, upload=272678]: Long tail cancellation due to optimal threshold Dec 18 14:26:48.541046764 (310h57m48.412129809s) [noise=true , address=77.90.31.11:4556 , piece_id=4GEBIFAQJYJX4KTPHTYUODQZ6ABTTPOA5SYOJBGUV7YDFIHHVO2Q, seg_pos={0 0}, upload=272678]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: expected piece hash; context canceled; EOF Dec 18 14:26:48.541055401 (310h57m48.412138436s) [noise=true , address=77.90.31.11:4556 , piece_id=4GEBIFAQJYJX4KTPHTYUODQZ6ABTTPOA5SYOJBGUV7YDFIHHVO2Q, seg_pos={0 0}, upload=272678]: Long tail cancellation due to optimal threshold Dec 18 14:26:48.541086550 (310h57m48.412169595s) [seg_pos={0 0}, upload=272678]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 18 14:26:48.541155190 (310h57m48.412238226s) [upload=272678]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 18 14:26:48.541159900 (310h57m48.412242935s) [upload=272678]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 18 14:26:48.836195919 (310h57m48.707278954s) [upload=272678]: Done uploading segments... Dec 18 14:26:48.836215697 (310h57m48.707298742s) [upload=272678]: Upload finished. err: <nil> Duration: 6.206454411s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [......................X....................X................] duration: 6.07179177s trace: https://jaeger.dev.storj.io/trace/2cbc07c217c30b3f FAIL (2025-12-18 14:40:08 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, Piece ID: O7IY6V6T4UILG3SXQ6K3SPXBVA2VLB22PWVQYIYORGLLYN6LE64A) context deadline exceeded; piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: I2SLWBEKFHZXWYTBZF73DICAZ4ZBXQYU6DES2QWLJMUCFZOCC7CA) context deadline exceeded ======LOGS====== ec 18 14:26:45.953557484 (310h57m45.824640529s) [upload=272678]: Starting upload Dec 18 14:26:45.953631205 (310h57m45.82471425s) [upload=272678]: Done starting upload Dec 18 14:26:45.953727147 (310h57m45.824810202s) [upload=272678]: Uploading segments... Dec 18 14:26:45.953920536 (310h57m45.825003581s) [upload=272678]: Got next segment. Inline: false Dec 18 14:26:45.953955743 (310h57m45.825038798s) [upload=272678]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 18 14:26:45.953963277 (310h57m45.825046322s) [upload=272678]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 18 14:26:46.201381593 (310h57m46.072464628s) [seg_pos={0 0}, upload=272678]: Begin upload segment... Dec 18 14:26:46.201430356 (310h57m46.072513391s) [seg_pos={0 0}, upload=272678]: Done begin upload segment. Dec 18 14:26:46.201523023 (310h57m46.072606088s) [noise=true , address=145.79.227.45:4231 , piece_id=V7B6RPI74SKRJQZCUWFHB2ECD46T3V224VKBL47NIIK5YH6PJCUQ, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.201534584 (310h57m46.072617619s) [noise=true , address=45.91.116.8:4048 , piece_id=BHQYLAJFKTBQ2WGY4XGRI4UOPTGB3SFEUVYZ7XWAC2BKA3ISU3WA, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.203237186 (310h57m46.074320231s) [noise=true , address=45.91.116.14:4190 , piece_id=I2SLWBEKFHZXWYTBZF73DICAZ4ZBXQYU6DES2QWLJMUCFZOCC7CA, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.203669961 (310h57m46.074752996s) [noise=true , address=145.79.227.36:4519 , piece_id=3E54JIZJCR55H36BEOLTDKWRMYA3WHNDPBSALSMNRY45O3BYO6JA, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.203970172 (310h57m46.075053207s) [noise=true , address=145.79.227.38:4978 , piece_id=FRWJ6Z3EH45YRMQVU235FJB75JWK7HOCLZXS7ASITOHLTL3YY5PQ, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.204250205 (310h57m46.07533324s) [noise=true , address=45.91.116.10:4084 , piece_id=UOTQVSUYETMA7UOEURUAT5RPX4QUO4FOXSLQY7UDUKJBCYCOH5BA, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.204534577 (310h57m46.075617612s) [noise=true , address=45.91.116.5:4299 , piece_id=SIZBVTWHVPXBL7OMRRQ467KFLRB56JNA5WDRNSXSNHPSV7UYG5LQ, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.204821995 (310h57m46.07590504s) [noise=true , address=145.79.227.29:4943 , piece_id=4DWWCYWOAU7NSESXDCONC6THXO3TTUYJMN7G6PTG2CP2OOOY2USA, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.206113864 (310h57m46.0771969s) [noise=true , address=77.90.31.11:4556 , piece_id=4GEBIFAQJYJX4KTPHTYUODQZ6ABTTPOA5SYOJBGUV7YDFIHHVO2Q, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.206151877 (310h57m46.077234912s) [noise=true , address=145.79.227.3:4150 , piece_id=XR5E35XW6KBFTREM2GFHARKAAIAINOWMLVYDKKBPFQBXJZOFCC2Q, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.207752204 (310h57m46.078835239s) [noise=true , address=145.79.227.39:4013 , piece_id=2SYWMQU5R7MCTH2E4X732AOVIRVPGUNNXLIN3UFCU7WF4BEJUACQ, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.210366552 (310h57m46.081449597s) [noise=true , address=145.79.227.35:4482 , piece_id=3EGA2CLZBLJWSSULQCBLEEHCMCJDE2FSOY4FCANAQXPEYTWIF53A, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.211613657 (310h57m46.082696692s) [noise=true , address=145.79.227.47:4266 , piece_id=4444BNJTIBUQWLNGTEK3IFD4TIWTCIBCWUHSYBMD3FGZSNDPWT3Q, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.212828561 (310h57m46.083911596s) [noise=true , address=145.79.227.41:4086 , piece_id=DFSNBYWUJ6Z65VTU5ME2IGEDB6KLMTAJM65RC44FUVYHTPEW4ZSA, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.215707031 (310h57m46.086790066s) [noise=true , address=45.91.116.13:4155 , piece_id=O7IY6V6T4UILG3SXQ6K3SPXBVA2VLB22PWVQYIYORGLLYN6LE64A, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.216889753 (310h57m46.087972798s) [noise=true , address=145.79.227.31:4797 , piece_id=F5WHLSRZSSZ4CRJRR2ZPYGGG67SHQ3QMAIHIC42NOVDW2UXARINQ, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:47.912125819 (310h57m47.783208865s) [upload=272678]: Next returned nil segment Dec 18 14:26:47.912143884 (310h57m47.783226929s) [upload=272678]: Waiting for error group managing segments... Dec 18 14:26:48.193212096 (310h57m48.064295131s) [noise=true , address=145.79.227.45:4231 , piece_id=V7B6RPI74SKRJQZCUWFHB2ECD46T3V224VKBL47NIIK5YH6PJCUQ, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.207387668 (310h57m48.078470703s) [noise=true , address=145.79.227.31:4797 , piece_id=F5WHLSRZSSZ4CRJRR2ZPYGGG67SHQ3QMAIHIC42NOVDW2UXARINQ, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.219064714 (310h57m48.090147759s) [noise=true , address=145.79.227.39:4013 , piece_id=2SYWMQU5R7MCTH2E4X732AOVIRVPGUNNXLIN3UFCU7WF4BEJUACQ, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.227887805 (310h57m48.09897085s) [noise=true , address=145.79.227.36:4519 , piece_id=3E54JIZJCR55H36BEOLTDKWRMYA3WHNDPBSALSMNRY45O3BYO6JA, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.229044407 (310h57m48.100127442s) [noise=true , address=145.79.227.41:4086 , piece_id=DFSNBYWUJ6Z65VTU5ME2IGEDB6KLMTAJM65RC44FUVYHTPEW4ZSA, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.234560812 (310h57m48.105643857s) [noise=true , address=145.79.227.47:4266 , piece_id=4444BNJTIBUQWLNGTEK3IFD4TIWTCIBCWUHSYBMD3FGZSNDPWT3Q, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.237048037 (310h57m48.108131072s) [noise=true , address=145.79.227.35:4482 , piece_id=3EGA2CLZBLJWSSULQCBLEEHCMCJDE2FSOY4FCANAQXPEYTWIF53A, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.267241984 (310h57m48.138325019s) [noise=true , address=145.79.227.29:4943 , piece_id=4DWWCYWOAU7NSESXDCONC6THXO3TTUYJMN7G6PTG2CP2OOOY2USA, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.288475576 (310h57m48.159558621s) [noise=true , address=145.79.227.3:4150 , piece_id=XR5E35XW6KBFTREM2GFHARKAAIAINOWMLVYDKKBPFQBXJZOFCC2Q, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.293478393 (310h57m48.164561458s) [noise=true , address=145.79.227.38:4978 , piece_id=FRWJ6Z3EH45YRMQVU235FJB75JWK7HOCLZXS7ASITOHLTL3YY5PQ, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.423759785 (310h57m48.29484282s) [noise=true , address=45.91.116.14:4190 , piece_id=I2SLWBEKFHZXWYTBZF73DICAZ4ZBXQYU6DES2QWLJMUCFZOCC7CA, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.485697347 (310h57m48.356780392s) [noise=true , address=45.91.116.10:4084 , piece_id=UOTQVSUYETMA7UOEURUAT5RPX4QUO4FOXSLQY7UDUKJBCYCOH5BA, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.540540711 (310h57m48.411623756s) [noise=true , address=45.91.116.13:4155 , piece_id=O7IY6V6T4UILG3SXQ6K3SPXBVA2VLB22PWVQYIYORGLLYN6LE64A, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.540575787 (310h57m48.411658832s) [seg_pos={0 0}, upload=272678]: Segment reached optimal threshold of 13 pieces. Dec 18 14:26:48.540831424 (310h57m48.411914459s) [noise=true , address=45.91.116.8:4048 , piece_id=BHQYLAJFKTBQ2WGY4XGRI4UOPTGB3SFEUVYZ7XWAC2BKA3ISU3WA, seg_pos={0 0}, upload=272678]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: write tcp 188.166.225.170:49328->45.91.116.8:4048: use of closed network connection; write tcp 188.166.225.170:49328->45.91.116.8:4048: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49328->45.91.116.8:4048: use of closed network connection Dec 18 14:26:48.540840511 (310h57m48.411923556s) [noise=true , address=45.91.116.8:4048 , piece_id=BHQYLAJFKTBQ2WGY4XGRI4UOPTGB3SFEUVYZ7XWAC2BKA3ISU3WA, seg_pos={0 0}, upload=272678]: Long tail cancellation due to optimal threshold Dec 18 14:26:48.540888052 (310h57m48.411971107s) [noise=true , address=45.91.116.5:4299 , piece_id=SIZBVTWHVPXBL7OMRRQ467KFLRB56JNA5WDRNSXSNHPSV7UYG5LQ, seg_pos={0 0}, upload=272678]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Dec 18 14:26:48.540899453 (310h57m48.411982498s) [noise=true , address=45.91.116.5:4299 , piece_id=SIZBVTWHVPXBL7OMRRQ467KFLRB56JNA5WDRNSXSNHPSV7UYG5LQ, seg_pos={0 0}, upload=272678]: Long tail cancellation due to optimal threshold Dec 18 14:26:48.541046764 (310h57m48.412129809s) [noise=true , address=77.90.31.11:4556 , piece_id=4GEBIFAQJYJX4KTPHTYUODQZ6ABTTPOA5SYOJBGUV7YDFIHHVO2Q, seg_pos={0 0}, upload=272678]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: expected piece hash; context canceled; EOF Dec 18 14:26:48.541055401 (310h57m48.412138436s) [noise=true , address=77.90.31.11:4556 , piece_id=4GEBIFAQJYJX4KTPHTYUODQZ6ABTTPOA5SYOJBGUV7YDFIHHVO2Q, seg_pos={0 0}, upload=272678]: Long tail cancellation due to optimal threshold Dec 18 14:26:48.541086550 (310h57m48.412169595s) [seg_pos={0 0}, upload=272678]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 18 14:26:48.541155190 (310h57m48.412238226s) [upload=272678]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 18 14:26:48.541159900 (310h57m48.412242935s) [upload=272678]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 18 14:26:48.836195919 (310h57m48.707278954s) [upload=272678]: Done uploading segments... Dec 18 14:26:48.836215697 (310h57m48.707298742s) [upload=272678]: Upload finished. err: <nil> Duration: 6.071842476s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 20 out of 60 is failed (threshold=6) [..............X.......X.....X.X.XX...XXXXX.XX..XX.XXXX.....X] duration: 5.001576341s trace: https://jaeger.dev.storj.io/trace/142285b66b9055cd FAIL (2025-12-18 14:39:43 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001609774s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......................X....................X...............] duration: 6.008120132s trace: https://jaeger.dev.storj.io/trace/2cbc07c217c30b3f FAIL (2025-12-18 14:38:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, Piece ID: O7IY6V6T4UILG3SXQ6K3SPXBVA2VLB22PWVQYIYORGLLYN6LE64A) context deadline exceeded; piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: I2SLWBEKFHZXWYTBZF73DICAZ4ZBXQYU6DES2QWLJMUCFZOCC7CA) context deadline exceeded ======LOGS====== ec 18 14:26:45.953557484 (310h57m45.824640529s) [upload=272678]: Starting upload Dec 18 14:26:45.953631205 (310h57m45.82471425s) [upload=272678]: Done starting upload Dec 18 14:26:45.953727147 (310h57m45.824810202s) [upload=272678]: Uploading segments... Dec 18 14:26:45.953920536 (310h57m45.825003581s) [upload=272678]: Got next segment. Inline: false Dec 18 14:26:45.953955743 (310h57m45.825038798s) [upload=272678]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 18 14:26:45.953963277 (310h57m45.825046322s) [upload=272678]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 18 14:26:46.201381593 (310h57m46.072464628s) [seg_pos={0 0}, upload=272678]: Begin upload segment... Dec 18 14:26:46.201430356 (310h57m46.072513391s) [seg_pos={0 0}, upload=272678]: Done begin upload segment. Dec 18 14:26:46.201523023 (310h57m46.072606088s) [noise=true , address=145.79.227.45:4231 , piece_id=V7B6RPI74SKRJQZCUWFHB2ECD46T3V224VKBL47NIIK5YH6PJCUQ, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.201534584 (310h57m46.072617619s) [noise=true , address=45.91.116.8:4048 , piece_id=BHQYLAJFKTBQ2WGY4XGRI4UOPTGB3SFEUVYZ7XWAC2BKA3ISU3WA, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.203237186 (310h57m46.074320231s) [noise=true , address=45.91.116.14:4190 , piece_id=I2SLWBEKFHZXWYTBZF73DICAZ4ZBXQYU6DES2QWLJMUCFZOCC7CA, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.203669961 (310h57m46.074752996s) [noise=true , address=145.79.227.36:4519 , piece_id=3E54JIZJCR55H36BEOLTDKWRMYA3WHNDPBSALSMNRY45O3BYO6JA, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.203970172 (310h57m46.075053207s) [noise=true , address=145.79.227.38:4978 , piece_id=FRWJ6Z3EH45YRMQVU235FJB75JWK7HOCLZXS7ASITOHLTL3YY5PQ, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.204250205 (310h57m46.07533324s) [noise=true , address=45.91.116.10:4084 , piece_id=UOTQVSUYETMA7UOEURUAT5RPX4QUO4FOXSLQY7UDUKJBCYCOH5BA, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.204534577 (310h57m46.075617612s) [noise=true , address=45.91.116.5:4299 , piece_id=SIZBVTWHVPXBL7OMRRQ467KFLRB56JNA5WDRNSXSNHPSV7UYG5LQ, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.204821995 (310h57m46.07590504s) [noise=true , address=145.79.227.29:4943 , piece_id=4DWWCYWOAU7NSESXDCONC6THXO3TTUYJMN7G6PTG2CP2OOOY2USA, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.206113864 (310h57m46.0771969s) [noise=true , address=77.90.31.11:4556 , piece_id=4GEBIFAQJYJX4KTPHTYUODQZ6ABTTPOA5SYOJBGUV7YDFIHHVO2Q, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.206151877 (310h57m46.077234912s) [noise=true , address=145.79.227.3:4150 , piece_id=XR5E35XW6KBFTREM2GFHARKAAIAINOWMLVYDKKBPFQBXJZOFCC2Q, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.207752204 (310h57m46.078835239s) [noise=true , address=145.79.227.39:4013 , piece_id=2SYWMQU5R7MCTH2E4X732AOVIRVPGUNNXLIN3UFCU7WF4BEJUACQ, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.210366552 (310h57m46.081449597s) [noise=true , address=145.79.227.35:4482 , piece_id=3EGA2CLZBLJWSSULQCBLEEHCMCJDE2FSOY4FCANAQXPEYTWIF53A, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.211613657 (310h57m46.082696692s) [noise=true , address=145.79.227.47:4266 , piece_id=4444BNJTIBUQWLNGTEK3IFD4TIWTCIBCWUHSYBMD3FGZSNDPWT3Q, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.212828561 (310h57m46.083911596s) [noise=true , address=145.79.227.41:4086 , piece_id=DFSNBYWUJ6Z65VTU5ME2IGEDB6KLMTAJM65RC44FUVYHTPEW4ZSA, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.215707031 (310h57m46.086790066s) [noise=true , address=45.91.116.13:4155 , piece_id=O7IY6V6T4UILG3SXQ6K3SPXBVA2VLB22PWVQYIYORGLLYN6LE64A, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.216889753 (310h57m46.087972798s) [noise=true , address=145.79.227.31:4797 , piece_id=F5WHLSRZSSZ4CRJRR2ZPYGGG67SHQ3QMAIHIC42NOVDW2UXARINQ, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:47.912125819 (310h57m47.783208865s) [upload=272678]: Next returned nil segment Dec 18 14:26:47.912143884 (310h57m47.783226929s) [upload=272678]: Waiting for error group managing segments... Dec 18 14:26:48.193212096 (310h57m48.064295131s) [noise=true , address=145.79.227.45:4231 , piece_id=V7B6RPI74SKRJQZCUWFHB2ECD46T3V224VKBL47NIIK5YH6PJCUQ, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.207387668 (310h57m48.078470703s) [noise=true , address=145.79.227.31:4797 , piece_id=F5WHLSRZSSZ4CRJRR2ZPYGGG67SHQ3QMAIHIC42NOVDW2UXARINQ, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.219064714 (310h57m48.090147759s) [noise=true , address=145.79.227.39:4013 , piece_id=2SYWMQU5R7MCTH2E4X732AOVIRVPGUNNXLIN3UFCU7WF4BEJUACQ, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.227887805 (310h57m48.09897085s) [noise=true , address=145.79.227.36:4519 , piece_id=3E54JIZJCR55H36BEOLTDKWRMYA3WHNDPBSALSMNRY45O3BYO6JA, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.229044407 (310h57m48.100127442s) [noise=true , address=145.79.227.41:4086 , piece_id=DFSNBYWUJ6Z65VTU5ME2IGEDB6KLMTAJM65RC44FUVYHTPEW4ZSA, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.234560812 (310h57m48.105643857s) [noise=true , address=145.79.227.47:4266 , piece_id=4444BNJTIBUQWLNGTEK3IFD4TIWTCIBCWUHSYBMD3FGZSNDPWT3Q, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.237048037 (310h57m48.108131072s) [noise=true , address=145.79.227.35:4482 , piece_id=3EGA2CLZBLJWSSULQCBLEEHCMCJDE2FSOY4FCANAQXPEYTWIF53A, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.267241984 (310h57m48.138325019s) [noise=true , address=145.79.227.29:4943 , piece_id=4DWWCYWOAU7NSESXDCONC6THXO3TTUYJMN7G6PTG2CP2OOOY2USA, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.288475576 (310h57m48.159558621s) [noise=true , address=145.79.227.3:4150 , piece_id=XR5E35XW6KBFTREM2GFHARKAAIAINOWMLVYDKKBPFQBXJZOFCC2Q, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.293478393 (310h57m48.164561458s) [noise=true , address=145.79.227.38:4978 , piece_id=FRWJ6Z3EH45YRMQVU235FJB75JWK7HOCLZXS7ASITOHLTL3YY5PQ, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.423759785 (310h57m48.29484282s) [noise=true , address=45.91.116.14:4190 , piece_id=I2SLWBEKFHZXWYTBZF73DICAZ4ZBXQYU6DES2QWLJMUCFZOCC7CA, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.485697347 (310h57m48.356780392s) [noise=true , address=45.91.116.10:4084 , piece_id=UOTQVSUYETMA7UOEURUAT5RPX4QUO4FOXSLQY7UDUKJBCYCOH5BA, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.540540711 (310h57m48.411623756s) [noise=true , address=45.91.116.13:4155 , piece_id=O7IY6V6T4UILG3SXQ6K3SPXBVA2VLB22PWVQYIYORGLLYN6LE64A, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.540575787 (310h57m48.411658832s) [seg_pos={0 0}, upload=272678]: Segment reached optimal threshold of 13 pieces. Dec 18 14:26:48.540831424 (310h57m48.411914459s) [noise=true , address=45.91.116.8:4048 , piece_id=BHQYLAJFKTBQ2WGY4XGRI4UOPTGB3SFEUVYZ7XWAC2BKA3ISU3WA, seg_pos={0 0}, upload=272678]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: write tcp 188.166.225.170:49328->45.91.116.8:4048: use of closed network connection; write tcp 188.166.225.170:49328->45.91.116.8:4048: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49328->45.91.116.8:4048: use of closed network connection Dec 18 14:26:48.540840511 (310h57m48.411923556s) [noise=true , address=45.91.116.8:4048 , piece_id=BHQYLAJFKTBQ2WGY4XGRI4UOPTGB3SFEUVYZ7XWAC2BKA3ISU3WA, seg_pos={0 0}, upload=272678]: Long tail cancellation due to optimal threshold Dec 18 14:26:48.540888052 (310h57m48.411971107s) [noise=true , address=45.91.116.5:4299 , piece_id=SIZBVTWHVPXBL7OMRRQ467KFLRB56JNA5WDRNSXSNHPSV7UYG5LQ, seg_pos={0 0}, upload=272678]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Dec 18 14:26:48.540899453 (310h57m48.411982498s) [noise=true , address=45.91.116.5:4299 , piece_id=SIZBVTWHVPXBL7OMRRQ467KFLRB56JNA5WDRNSXSNHPSV7UYG5LQ, seg_pos={0 0}, upload=272678]: Long tail cancellation due to optimal threshold Dec 18 14:26:48.541046764 (310h57m48.412129809s) [noise=true , address=77.90.31.11:4556 , piece_id=4GEBIFAQJYJX4KTPHTYUODQZ6ABTTPOA5SYOJBGUV7YDFIHHVO2Q, seg_pos={0 0}, upload=272678]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: expected piece hash; context canceled; EOF Dec 18 14:26:48.541055401 (310h57m48.412138436s) [noise=true , address=77.90.31.11:4556 , piece_id=4GEBIFAQJYJX4KTPHTYUODQZ6ABTTPOA5SYOJBGUV7YDFIHHVO2Q, seg_pos={0 0}, upload=272678]: Long tail cancellation due to optimal threshold Dec 18 14:26:48.541086550 (310h57m48.412169595s) [seg_pos={0 0}, upload=272678]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 18 14:26:48.541155190 (310h57m48.412238226s) [upload=272678]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 18 14:26:48.541159900 (310h57m48.412242935s) [upload=272678]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 18 14:26:48.836195919 (310h57m48.707278954s) [upload=272678]: Done uploading segments... Dec 18 14:26:48.836215697 (310h57m48.707298742s) [upload=272678]: Upload finished. err: <nil> Duration: 6.008144287s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 20 out of 60 is failed (threshold=6) [X..............X.......X.....X.X.XX...XXXXX.XX..XX.XXXX.....] duration: 5.00132033s trace: https://jaeger.dev.storj.io/trace/9dbfc1dffabb5ea FAIL (2025-12-18 14:38:28 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001346119s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [........................X....................X..............] duration: 5.738339794s trace: https://jaeger.dev.storj.io/trace/2cbc07c217c30b3f FAIL (2025-12-18 14:37:58 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, Piece ID: O7IY6V6T4UILG3SXQ6K3SPXBVA2VLB22PWVQYIYORGLLYN6LE64A) context deadline exceeded; piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: I2SLWBEKFHZXWYTBZF73DICAZ4ZBXQYU6DES2QWLJMUCFZOCC7CA) context deadline exceeded ======LOGS====== ec 18 14:26:45.953557484 (310h57m45.824640529s) [upload=272678]: Starting upload Dec 18 14:26:45.953631205 (310h57m45.82471425s) [upload=272678]: Done starting upload Dec 18 14:26:45.953727147 (310h57m45.824810202s) [upload=272678]: Uploading segments... Dec 18 14:26:45.953920536 (310h57m45.825003581s) [upload=272678]: Got next segment. Inline: false Dec 18 14:26:45.953955743 (310h57m45.825038798s) [upload=272678]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 18 14:26:45.953963277 (310h57m45.825046322s) [upload=272678]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 18 14:26:46.201381593 (310h57m46.072464628s) [seg_pos={0 0}, upload=272678]: Begin upload segment... Dec 18 14:26:46.201430356 (310h57m46.072513391s) [seg_pos={0 0}, upload=272678]: Done begin upload segment. Dec 18 14:26:46.201523023 (310h57m46.072606088s) [noise=true , address=145.79.227.45:4231 , piece_id=V7B6RPI74SKRJQZCUWFHB2ECD46T3V224VKBL47NIIK5YH6PJCUQ, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.201534584 (310h57m46.072617619s) [noise=true , address=45.91.116.8:4048 , piece_id=BHQYLAJFKTBQ2WGY4XGRI4UOPTGB3SFEUVYZ7XWAC2BKA3ISU3WA, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.203237186 (310h57m46.074320231s) [noise=true , address=45.91.116.14:4190 , piece_id=I2SLWBEKFHZXWYTBZF73DICAZ4ZBXQYU6DES2QWLJMUCFZOCC7CA, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.203669961 (310h57m46.074752996s) [noise=true , address=145.79.227.36:4519 , piece_id=3E54JIZJCR55H36BEOLTDKWRMYA3WHNDPBSALSMNRY45O3BYO6JA, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.203970172 (310h57m46.075053207s) [noise=true , address=145.79.227.38:4978 , piece_id=FRWJ6Z3EH45YRMQVU235FJB75JWK7HOCLZXS7ASITOHLTL3YY5PQ, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.204250205 (310h57m46.07533324s) [noise=true , address=45.91.116.10:4084 , piece_id=UOTQVSUYETMA7UOEURUAT5RPX4QUO4FOXSLQY7UDUKJBCYCOH5BA, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.204534577 (310h57m46.075617612s) [noise=true , address=45.91.116.5:4299 , piece_id=SIZBVTWHVPXBL7OMRRQ467KFLRB56JNA5WDRNSXSNHPSV7UYG5LQ, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.204821995 (310h57m46.07590504s) [noise=true , address=145.79.227.29:4943 , piece_id=4DWWCYWOAU7NSESXDCONC6THXO3TTUYJMN7G6PTG2CP2OOOY2USA, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.206113864 (310h57m46.0771969s) [noise=true , address=77.90.31.11:4556 , piece_id=4GEBIFAQJYJX4KTPHTYUODQZ6ABTTPOA5SYOJBGUV7YDFIHHVO2Q, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.206151877 (310h57m46.077234912s) [noise=true , address=145.79.227.3:4150 , piece_id=XR5E35XW6KBFTREM2GFHARKAAIAINOWMLVYDKKBPFQBXJZOFCC2Q, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.207752204 (310h57m46.078835239s) [noise=true , address=145.79.227.39:4013 , piece_id=2SYWMQU5R7MCTH2E4X732AOVIRVPGUNNXLIN3UFCU7WF4BEJUACQ, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.210366552 (310h57m46.081449597s) [noise=true , address=145.79.227.35:4482 , piece_id=3EGA2CLZBLJWSSULQCBLEEHCMCJDE2FSOY4FCANAQXPEYTWIF53A, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.211613657 (310h57m46.082696692s) [noise=true , address=145.79.227.47:4266 , piece_id=4444BNJTIBUQWLNGTEK3IFD4TIWTCIBCWUHSYBMD3FGZSNDPWT3Q, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.212828561 (310h57m46.083911596s) [noise=true , address=145.79.227.41:4086 , piece_id=DFSNBYWUJ6Z65VTU5ME2IGEDB6KLMTAJM65RC44FUVYHTPEW4ZSA, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.215707031 (310h57m46.086790066s) [noise=true , address=45.91.116.13:4155 , piece_id=O7IY6V6T4UILG3SXQ6K3SPXBVA2VLB22PWVQYIYORGLLYN6LE64A, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.216889753 (310h57m46.087972798s) [noise=true , address=145.79.227.31:4797 , piece_id=F5WHLSRZSSZ4CRJRR2ZPYGGG67SHQ3QMAIHIC42NOVDW2UXARINQ, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:47.912125819 (310h57m47.783208865s) [upload=272678]: Next returned nil segment Dec 18 14:26:47.912143884 (310h57m47.783226929s) [upload=272678]: Waiting for error group managing segments... Dec 18 14:26:48.193212096 (310h57m48.064295131s) [noise=true , address=145.79.227.45:4231 , piece_id=V7B6RPI74SKRJQZCUWFHB2ECD46T3V224VKBL47NIIK5YH6PJCUQ, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.207387668 (310h57m48.078470703s) [noise=true , address=145.79.227.31:4797 , piece_id=F5WHLSRZSSZ4CRJRR2ZPYGGG67SHQ3QMAIHIC42NOVDW2UXARINQ, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.219064714 (310h57m48.090147759s) [noise=true , address=145.79.227.39:4013 , piece_id=2SYWMQU5R7MCTH2E4X732AOVIRVPGUNNXLIN3UFCU7WF4BEJUACQ, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.227887805 (310h57m48.09897085s) [noise=true , address=145.79.227.36:4519 , piece_id=3E54JIZJCR55H36BEOLTDKWRMYA3WHNDPBSALSMNRY45O3BYO6JA, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.229044407 (310h57m48.100127442s) [noise=true , address=145.79.227.41:4086 , piece_id=DFSNBYWUJ6Z65VTU5ME2IGEDB6KLMTAJM65RC44FUVYHTPEW4ZSA, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.234560812 (310h57m48.105643857s) [noise=true , address=145.79.227.47:4266 , piece_id=4444BNJTIBUQWLNGTEK3IFD4TIWTCIBCWUHSYBMD3FGZSNDPWT3Q, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.237048037 (310h57m48.108131072s) [noise=true , address=145.79.227.35:4482 , piece_id=3EGA2CLZBLJWSSULQCBLEEHCMCJDE2FSOY4FCANAQXPEYTWIF53A, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.267241984 (310h57m48.138325019s) [noise=true , address=145.79.227.29:4943 , piece_id=4DWWCYWOAU7NSESXDCONC6THXO3TTUYJMN7G6PTG2CP2OOOY2USA, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.288475576 (310h57m48.159558621s) [noise=true , address=145.79.227.3:4150 , piece_id=XR5E35XW6KBFTREM2GFHARKAAIAINOWMLVYDKKBPFQBXJZOFCC2Q, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.293478393 (310h57m48.164561458s) [noise=true , address=145.79.227.38:4978 , piece_id=FRWJ6Z3EH45YRMQVU235FJB75JWK7HOCLZXS7ASITOHLTL3YY5PQ, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.423759785 (310h57m48.29484282s) [noise=true , address=45.91.116.14:4190 , piece_id=I2SLWBEKFHZXWYTBZF73DICAZ4ZBXQYU6DES2QWLJMUCFZOCC7CA, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.485697347 (310h57m48.356780392s) [noise=true , address=45.91.116.10:4084 , piece_id=UOTQVSUYETMA7UOEURUAT5RPX4QUO4FOXSLQY7UDUKJBCYCOH5BA, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.540540711 (310h57m48.411623756s) [noise=true , address=45.91.116.13:4155 , piece_id=O7IY6V6T4UILG3SXQ6K3SPXBVA2VLB22PWVQYIYORGLLYN6LE64A, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.540575787 (310h57m48.411658832s) [seg_pos={0 0}, upload=272678]: Segment reached optimal threshold of 13 pieces. Dec 18 14:26:48.540831424 (310h57m48.411914459s) [noise=true , address=45.91.116.8:4048 , piece_id=BHQYLAJFKTBQ2WGY4XGRI4UOPTGB3SFEUVYZ7XWAC2BKA3ISU3WA, seg_pos={0 0}, upload=272678]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: write tcp 188.166.225.170:49328->45.91.116.8:4048: use of closed network connection; write tcp 188.166.225.170:49328->45.91.116.8:4048: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49328->45.91.116.8:4048: use of closed network connection Dec 18 14:26:48.540840511 (310h57m48.411923556s) [noise=true , address=45.91.116.8:4048 , piece_id=BHQYLAJFKTBQ2WGY4XGRI4UOPTGB3SFEUVYZ7XWAC2BKA3ISU3WA, seg_pos={0 0}, upload=272678]: Long tail cancellation due to optimal threshold Dec 18 14:26:48.540888052 (310h57m48.411971107s) [noise=true , address=45.91.116.5:4299 , piece_id=SIZBVTWHVPXBL7OMRRQ467KFLRB56JNA5WDRNSXSNHPSV7UYG5LQ, seg_pos={0 0}, upload=272678]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Dec 18 14:26:48.540899453 (310h57m48.411982498s) [noise=true , address=45.91.116.5:4299 , piece_id=SIZBVTWHVPXBL7OMRRQ467KFLRB56JNA5WDRNSXSNHPSV7UYG5LQ, seg_pos={0 0}, upload=272678]: Long tail cancellation due to optimal threshold Dec 18 14:26:48.541046764 (310h57m48.412129809s) [noise=true , address=77.90.31.11:4556 , piece_id=4GEBIFAQJYJX4KTPHTYUODQZ6ABTTPOA5SYOJBGUV7YDFIHHVO2Q, seg_pos={0 0}, upload=272678]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: expected piece hash; context canceled; EOF Dec 18 14:26:48.541055401 (310h57m48.412138436s) [noise=true , address=77.90.31.11:4556 , piece_id=4GEBIFAQJYJX4KTPHTYUODQZ6ABTTPOA5SYOJBGUV7YDFIHHVO2Q, seg_pos={0 0}, upload=272678]: Long tail cancellation due to optimal threshold Dec 18 14:26:48.541086550 (310h57m48.412169595s) [seg_pos={0 0}, upload=272678]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 18 14:26:48.541155190 (310h57m48.412238226s) [upload=272678]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 18 14:26:48.541159900 (310h57m48.412242935s) [upload=272678]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 18 14:26:48.836195919 (310h57m48.707278954s) [upload=272678]: Done uploading segments... Dec 18 14:26:48.836215697 (310h57m48.707298742s) [upload=272678]: Upload finished. err: <nil> Duration: 5.738370182s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 20 out of 60 is failed (threshold=6) [.X..............X.......X.....X.X.XX...XXXXX.XX..XX.XXXX....] duration: 3.607369799s trace: https://jaeger.dev.storj.io/trace/9dbfc1dffabb5ea FAIL (2025-12-18 14:37:30 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.607493454s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........................X....................X.............] duration: 6.116974142s trace: https://jaeger.dev.storj.io/trace/2cbc07c217c30b3f FAIL (2025-12-18 14:37:05 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, Piece ID: O7IY6V6T4UILG3SXQ6K3SPXBVA2VLB22PWVQYIYORGLLYN6LE64A) context deadline exceeded; piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: I2SLWBEKFHZXWYTBZF73DICAZ4ZBXQYU6DES2QWLJMUCFZOCC7CA) context deadline exceeded ======LOGS====== ec 18 14:26:45.953557484 (310h57m45.824640529s) [upload=272678]: Starting upload Dec 18 14:26:45.953631205 (310h57m45.82471425s) [upload=272678]: Done starting upload Dec 18 14:26:45.953727147 (310h57m45.824810202s) [upload=272678]: Uploading segments... Dec 18 14:26:45.953920536 (310h57m45.825003581s) [upload=272678]: Got next segment. Inline: false Dec 18 14:26:45.953955743 (310h57m45.825038798s) [upload=272678]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 18 14:26:45.953963277 (310h57m45.825046322s) [upload=272678]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 18 14:26:46.201381593 (310h57m46.072464628s) [seg_pos={0 0}, upload=272678]: Begin upload segment... Dec 18 14:26:46.201430356 (310h57m46.072513391s) [seg_pos={0 0}, upload=272678]: Done begin upload segment. Dec 18 14:26:46.201523023 (310h57m46.072606088s) [noise=true , address=145.79.227.45:4231 , piece_id=V7B6RPI74SKRJQZCUWFHB2ECD46T3V224VKBL47NIIK5YH6PJCUQ, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.201534584 (310h57m46.072617619s) [noise=true , address=45.91.116.8:4048 , piece_id=BHQYLAJFKTBQ2WGY4XGRI4UOPTGB3SFEUVYZ7XWAC2BKA3ISU3WA, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.203237186 (310h57m46.074320231s) [noise=true , address=45.91.116.14:4190 , piece_id=I2SLWBEKFHZXWYTBZF73DICAZ4ZBXQYU6DES2QWLJMUCFZOCC7CA, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.203669961 (310h57m46.074752996s) [noise=true , address=145.79.227.36:4519 , piece_id=3E54JIZJCR55H36BEOLTDKWRMYA3WHNDPBSALSMNRY45O3BYO6JA, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.203970172 (310h57m46.075053207s) [noise=true , address=145.79.227.38:4978 , piece_id=FRWJ6Z3EH45YRMQVU235FJB75JWK7HOCLZXS7ASITOHLTL3YY5PQ, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.204250205 (310h57m46.07533324s) [noise=true , address=45.91.116.10:4084 , piece_id=UOTQVSUYETMA7UOEURUAT5RPX4QUO4FOXSLQY7UDUKJBCYCOH5BA, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.204534577 (310h57m46.075617612s) [noise=true , address=45.91.116.5:4299 , piece_id=SIZBVTWHVPXBL7OMRRQ467KFLRB56JNA5WDRNSXSNHPSV7UYG5LQ, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.204821995 (310h57m46.07590504s) [noise=true , address=145.79.227.29:4943 , piece_id=4DWWCYWOAU7NSESXDCONC6THXO3TTUYJMN7G6PTG2CP2OOOY2USA, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.206113864 (310h57m46.0771969s) [noise=true , address=77.90.31.11:4556 , piece_id=4GEBIFAQJYJX4KTPHTYUODQZ6ABTTPOA5SYOJBGUV7YDFIHHVO2Q, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.206151877 (310h57m46.077234912s) [noise=true , address=145.79.227.3:4150 , piece_id=XR5E35XW6KBFTREM2GFHARKAAIAINOWMLVYDKKBPFQBXJZOFCC2Q, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.207752204 (310h57m46.078835239s) [noise=true , address=145.79.227.39:4013 , piece_id=2SYWMQU5R7MCTH2E4X732AOVIRVPGUNNXLIN3UFCU7WF4BEJUACQ, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.210366552 (310h57m46.081449597s) [noise=true , address=145.79.227.35:4482 , piece_id=3EGA2CLZBLJWSSULQCBLEEHCMCJDE2FSOY4FCANAQXPEYTWIF53A, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.211613657 (310h57m46.082696692s) [noise=true , address=145.79.227.47:4266 , piece_id=4444BNJTIBUQWLNGTEK3IFD4TIWTCIBCWUHSYBMD3FGZSNDPWT3Q, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.212828561 (310h57m46.083911596s) [noise=true , address=145.79.227.41:4086 , piece_id=DFSNBYWUJ6Z65VTU5ME2IGEDB6KLMTAJM65RC44FUVYHTPEW4ZSA, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.215707031 (310h57m46.086790066s) [noise=true , address=45.91.116.13:4155 , piece_id=O7IY6V6T4UILG3SXQ6K3SPXBVA2VLB22PWVQYIYORGLLYN6LE64A, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.216889753 (310h57m46.087972798s) [noise=true , address=145.79.227.31:4797 , piece_id=F5WHLSRZSSZ4CRJRR2ZPYGGG67SHQ3QMAIHIC42NOVDW2UXARINQ, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:47.912125819 (310h57m47.783208865s) [upload=272678]: Next returned nil segment Dec 18 14:26:47.912143884 (310h57m47.783226929s) [upload=272678]: Waiting for error group managing segments... Dec 18 14:26:48.193212096 (310h57m48.064295131s) [noise=true , address=145.79.227.45:4231 , piece_id=V7B6RPI74SKRJQZCUWFHB2ECD46T3V224VKBL47NIIK5YH6PJCUQ, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.207387668 (310h57m48.078470703s) [noise=true , address=145.79.227.31:4797 , piece_id=F5WHLSRZSSZ4CRJRR2ZPYGGG67SHQ3QMAIHIC42NOVDW2UXARINQ, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.219064714 (310h57m48.090147759s) [noise=true , address=145.79.227.39:4013 , piece_id=2SYWMQU5R7MCTH2E4X732AOVIRVPGUNNXLIN3UFCU7WF4BEJUACQ, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.227887805 (310h57m48.09897085s) [noise=true , address=145.79.227.36:4519 , piece_id=3E54JIZJCR55H36BEOLTDKWRMYA3WHNDPBSALSMNRY45O3BYO6JA, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.229044407 (310h57m48.100127442s) [noise=true , address=145.79.227.41:4086 , piece_id=DFSNBYWUJ6Z65VTU5ME2IGEDB6KLMTAJM65RC44FUVYHTPEW4ZSA, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.234560812 (310h57m48.105643857s) [noise=true , address=145.79.227.47:4266 , piece_id=4444BNJTIBUQWLNGTEK3IFD4TIWTCIBCWUHSYBMD3FGZSNDPWT3Q, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.237048037 (310h57m48.108131072s) [noise=true , address=145.79.227.35:4482 , piece_id=3EGA2CLZBLJWSSULQCBLEEHCMCJDE2FSOY4FCANAQXPEYTWIF53A, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.267241984 (310h57m48.138325019s) [noise=true , address=145.79.227.29:4943 , piece_id=4DWWCYWOAU7NSESXDCONC6THXO3TTUYJMN7G6PTG2CP2OOOY2USA, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.288475576 (310h57m48.159558621s) [noise=true , address=145.79.227.3:4150 , piece_id=XR5E35XW6KBFTREM2GFHARKAAIAINOWMLVYDKKBPFQBXJZOFCC2Q, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.293478393 (310h57m48.164561458s) [noise=true , address=145.79.227.38:4978 , piece_id=FRWJ6Z3EH45YRMQVU235FJB75JWK7HOCLZXS7ASITOHLTL3YY5PQ, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.423759785 (310h57m48.29484282s) [noise=true , address=45.91.116.14:4190 , piece_id=I2SLWBEKFHZXWYTBZF73DICAZ4ZBXQYU6DES2QWLJMUCFZOCC7CA, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.485697347 (310h57m48.356780392s) [noise=true , address=45.91.116.10:4084 , piece_id=UOTQVSUYETMA7UOEURUAT5RPX4QUO4FOXSLQY7UDUKJBCYCOH5BA, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.540540711 (310h57m48.411623756s) [noise=true , address=45.91.116.13:4155 , piece_id=O7IY6V6T4UILG3SXQ6K3SPXBVA2VLB22PWVQYIYORGLLYN6LE64A, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.540575787 (310h57m48.411658832s) [seg_pos={0 0}, upload=272678]: Segment reached optimal threshold of 13 pieces. Dec 18 14:26:48.540831424 (310h57m48.411914459s) [noise=true , address=45.91.116.8:4048 , piece_id=BHQYLAJFKTBQ2WGY4XGRI4UOPTGB3SFEUVYZ7XWAC2BKA3ISU3WA, seg_pos={0 0}, upload=272678]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: write tcp 188.166.225.170:49328->45.91.116.8:4048: use of closed network connection; write tcp 188.166.225.170:49328->45.91.116.8:4048: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49328->45.91.116.8:4048: use of closed network connection Dec 18 14:26:48.540840511 (310h57m48.411923556s) [noise=true , address=45.91.116.8:4048 , piece_id=BHQYLAJFKTBQ2WGY4XGRI4UOPTGB3SFEUVYZ7XWAC2BKA3ISU3WA, seg_pos={0 0}, upload=272678]: Long tail cancellation due to optimal threshold Dec 18 14:26:48.540888052 (310h57m48.411971107s) [noise=true , address=45.91.116.5:4299 , piece_id=SIZBVTWHVPXBL7OMRRQ467KFLRB56JNA5WDRNSXSNHPSV7UYG5LQ, seg_pos={0 0}, upload=272678]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Dec 18 14:26:48.540899453 (310h57m48.411982498s) [noise=true , address=45.91.116.5:4299 , piece_id=SIZBVTWHVPXBL7OMRRQ467KFLRB56JNA5WDRNSXSNHPSV7UYG5LQ, seg_pos={0 0}, upload=272678]: Long tail cancellation due to optimal threshold Dec 18 14:26:48.541046764 (310h57m48.412129809s) [noise=true , address=77.90.31.11:4556 , piece_id=4GEBIFAQJYJX4KTPHTYUODQZ6ABTTPOA5SYOJBGUV7YDFIHHVO2Q, seg_pos={0 0}, upload=272678]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: expected piece hash; context canceled; EOF Dec 18 14:26:48.541055401 (310h57m48.412138436s) [noise=true , address=77.90.31.11:4556 , piece_id=4GEBIFAQJYJX4KTPHTYUODQZ6ABTTPOA5SYOJBGUV7YDFIHHVO2Q, seg_pos={0 0}, upload=272678]: Long tail cancellation due to optimal threshold Dec 18 14:26:48.541086550 (310h57m48.412169595s) [seg_pos={0 0}, upload=272678]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 18 14:26:48.541155190 (310h57m48.412238226s) [upload=272678]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 18 14:26:48.541159900 (310h57m48.412242935s) [upload=272678]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 18 14:26:48.836195919 (310h57m48.707278954s) [upload=272678]: Done uploading segments... Dec 18 14:26:48.836215697 (310h57m48.707298742s) [upload=272678]: Upload finished. err: <nil> Duration: 6.117001283s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........................X....................X............] duration: 5.82471042s trace: https://jaeger.dev.storj.io/trace/2cbc07c217c30b3f FAIL (2025-12-18 14:36:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, Piece ID: O7IY6V6T4UILG3SXQ6K3SPXBVA2VLB22PWVQYIYORGLLYN6LE64A) context deadline exceeded; piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: I2SLWBEKFHZXWYTBZF73DICAZ4ZBXQYU6DES2QWLJMUCFZOCC7CA) context deadline exceeded ======LOGS====== ec 18 14:26:45.953557484 (310h57m45.824640529s) [upload=272678]: Starting upload Dec 18 14:26:45.953631205 (310h57m45.82471425s) [upload=272678]: Done starting upload Dec 18 14:26:45.953727147 (310h57m45.824810202s) [upload=272678]: Uploading segments... Dec 18 14:26:45.953920536 (310h57m45.825003581s) [upload=272678]: Got next segment. Inline: false Dec 18 14:26:45.953955743 (310h57m45.825038798s) [upload=272678]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 18 14:26:45.953963277 (310h57m45.825046322s) [upload=272678]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 18 14:26:46.201381593 (310h57m46.072464628s) [seg_pos={0 0}, upload=272678]: Begin upload segment... Dec 18 14:26:46.201430356 (310h57m46.072513391s) [seg_pos={0 0}, upload=272678]: Done begin upload segment. Dec 18 14:26:46.201523023 (310h57m46.072606088s) [noise=true , address=145.79.227.45:4231 , piece_id=V7B6RPI74SKRJQZCUWFHB2ECD46T3V224VKBL47NIIK5YH6PJCUQ, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.201534584 (310h57m46.072617619s) [noise=true , address=45.91.116.8:4048 , piece_id=BHQYLAJFKTBQ2WGY4XGRI4UOPTGB3SFEUVYZ7XWAC2BKA3ISU3WA, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.203237186 (310h57m46.074320231s) [noise=true , address=45.91.116.14:4190 , piece_id=I2SLWBEKFHZXWYTBZF73DICAZ4ZBXQYU6DES2QWLJMUCFZOCC7CA, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.203669961 (310h57m46.074752996s) [noise=true , address=145.79.227.36:4519 , piece_id=3E54JIZJCR55H36BEOLTDKWRMYA3WHNDPBSALSMNRY45O3BYO6JA, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.203970172 (310h57m46.075053207s) [noise=true , address=145.79.227.38:4978 , piece_id=FRWJ6Z3EH45YRMQVU235FJB75JWK7HOCLZXS7ASITOHLTL3YY5PQ, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.204250205 (310h57m46.07533324s) [noise=true , address=45.91.116.10:4084 , piece_id=UOTQVSUYETMA7UOEURUAT5RPX4QUO4FOXSLQY7UDUKJBCYCOH5BA, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.204534577 (310h57m46.075617612s) [noise=true , address=45.91.116.5:4299 , piece_id=SIZBVTWHVPXBL7OMRRQ467KFLRB56JNA5WDRNSXSNHPSV7UYG5LQ, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.204821995 (310h57m46.07590504s) [noise=true , address=145.79.227.29:4943 , piece_id=4DWWCYWOAU7NSESXDCONC6THXO3TTUYJMN7G6PTG2CP2OOOY2USA, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.206113864 (310h57m46.0771969s) [noise=true , address=77.90.31.11:4556 , piece_id=4GEBIFAQJYJX4KTPHTYUODQZ6ABTTPOA5SYOJBGUV7YDFIHHVO2Q, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.206151877 (310h57m46.077234912s) [noise=true , address=145.79.227.3:4150 , piece_id=XR5E35XW6KBFTREM2GFHARKAAIAINOWMLVYDKKBPFQBXJZOFCC2Q, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.207752204 (310h57m46.078835239s) [noise=true , address=145.79.227.39:4013 , piece_id=2SYWMQU5R7MCTH2E4X732AOVIRVPGUNNXLIN3UFCU7WF4BEJUACQ, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.210366552 (310h57m46.081449597s) [noise=true , address=145.79.227.35:4482 , piece_id=3EGA2CLZBLJWSSULQCBLEEHCMCJDE2FSOY4FCANAQXPEYTWIF53A, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.211613657 (310h57m46.082696692s) [noise=true , address=145.79.227.47:4266 , piece_id=4444BNJTIBUQWLNGTEK3IFD4TIWTCIBCWUHSYBMD3FGZSNDPWT3Q, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.212828561 (310h57m46.083911596s) [noise=true , address=145.79.227.41:4086 , piece_id=DFSNBYWUJ6Z65VTU5ME2IGEDB6KLMTAJM65RC44FUVYHTPEW4ZSA, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.215707031 (310h57m46.086790066s) [noise=true , address=45.91.116.13:4155 , piece_id=O7IY6V6T4UILG3SXQ6K3SPXBVA2VLB22PWVQYIYORGLLYN6LE64A, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.216889753 (310h57m46.087972798s) [noise=true , address=145.79.227.31:4797 , piece_id=F5WHLSRZSSZ4CRJRR2ZPYGGG67SHQ3QMAIHIC42NOVDW2UXARINQ, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:47.912125819 (310h57m47.783208865s) [upload=272678]: Next returned nil segment Dec 18 14:26:47.912143884 (310h57m47.783226929s) [upload=272678]: Waiting for error group managing segments... Dec 18 14:26:48.193212096 (310h57m48.064295131s) [noise=true , address=145.79.227.45:4231 , piece_id=V7B6RPI74SKRJQZCUWFHB2ECD46T3V224VKBL47NIIK5YH6PJCUQ, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.207387668 (310h57m48.078470703s) [noise=true , address=145.79.227.31:4797 , piece_id=F5WHLSRZSSZ4CRJRR2ZPYGGG67SHQ3QMAIHIC42NOVDW2UXARINQ, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.219064714 (310h57m48.090147759s) [noise=true , address=145.79.227.39:4013 , piece_id=2SYWMQU5R7MCTH2E4X732AOVIRVPGUNNXLIN3UFCU7WF4BEJUACQ, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.227887805 (310h57m48.09897085s) [noise=true , address=145.79.227.36:4519 , piece_id=3E54JIZJCR55H36BEOLTDKWRMYA3WHNDPBSALSMNRY45O3BYO6JA, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.229044407 (310h57m48.100127442s) [noise=true , address=145.79.227.41:4086 , piece_id=DFSNBYWUJ6Z65VTU5ME2IGEDB6KLMTAJM65RC44FUVYHTPEW4ZSA, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.234560812 (310h57m48.105643857s) [noise=true , address=145.79.227.47:4266 , piece_id=4444BNJTIBUQWLNGTEK3IFD4TIWTCIBCWUHSYBMD3FGZSNDPWT3Q, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.237048037 (310h57m48.108131072s) [noise=true , address=145.79.227.35:4482 , piece_id=3EGA2CLZBLJWSSULQCBLEEHCMCJDE2FSOY4FCANAQXPEYTWIF53A, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.267241984 (310h57m48.138325019s) [noise=true , address=145.79.227.29:4943 , piece_id=4DWWCYWOAU7NSESXDCONC6THXO3TTUYJMN7G6PTG2CP2OOOY2USA, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.288475576 (310h57m48.159558621s) [noise=true , address=145.79.227.3:4150 , piece_id=XR5E35XW6KBFTREM2GFHARKAAIAINOWMLVYDKKBPFQBXJZOFCC2Q, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.293478393 (310h57m48.164561458s) [noise=true , address=145.79.227.38:4978 , piece_id=FRWJ6Z3EH45YRMQVU235FJB75JWK7HOCLZXS7ASITOHLTL3YY5PQ, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.423759785 (310h57m48.29484282s) [noise=true , address=45.91.116.14:4190 , piece_id=I2SLWBEKFHZXWYTBZF73DICAZ4ZBXQYU6DES2QWLJMUCFZOCC7CA, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.485697347 (310h57m48.356780392s) [noise=true , address=45.91.116.10:4084 , piece_id=UOTQVSUYETMA7UOEURUAT5RPX4QUO4FOXSLQY7UDUKJBCYCOH5BA, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.540540711 (310h57m48.411623756s) [noise=true , address=45.91.116.13:4155 , piece_id=O7IY6V6T4UILG3SXQ6K3SPXBVA2VLB22PWVQYIYORGLLYN6LE64A, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.540575787 (310h57m48.411658832s) [seg_pos={0 0}, upload=272678]: Segment reached optimal threshold of 13 pieces. Dec 18 14:26:48.540831424 (310h57m48.411914459s) [noise=true , address=45.91.116.8:4048 , piece_id=BHQYLAJFKTBQ2WGY4XGRI4UOPTGB3SFEUVYZ7XWAC2BKA3ISU3WA, seg_pos={0 0}, upload=272678]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: write tcp 188.166.225.170:49328->45.91.116.8:4048: use of closed network connection; write tcp 188.166.225.170:49328->45.91.116.8:4048: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49328->45.91.116.8:4048: use of closed network connection Dec 18 14:26:48.540840511 (310h57m48.411923556s) [noise=true , address=45.91.116.8:4048 , piece_id=BHQYLAJFKTBQ2WGY4XGRI4UOPTGB3SFEUVYZ7XWAC2BKA3ISU3WA, seg_pos={0 0}, upload=272678]: Long tail cancellation due to optimal threshold Dec 18 14:26:48.540888052 (310h57m48.411971107s) [noise=true , address=45.91.116.5:4299 , piece_id=SIZBVTWHVPXBL7OMRRQ467KFLRB56JNA5WDRNSXSNHPSV7UYG5LQ, seg_pos={0 0}, upload=272678]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Dec 18 14:26:48.540899453 (310h57m48.411982498s) [noise=true , address=45.91.116.5:4299 , piece_id=SIZBVTWHVPXBL7OMRRQ467KFLRB56JNA5WDRNSXSNHPSV7UYG5LQ, seg_pos={0 0}, upload=272678]: Long tail cancellation due to optimal threshold Dec 18 14:26:48.541046764 (310h57m48.412129809s) [noise=true , address=77.90.31.11:4556 , piece_id=4GEBIFAQJYJX4KTPHTYUODQZ6ABTTPOA5SYOJBGUV7YDFIHHVO2Q, seg_pos={0 0}, upload=272678]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: expected piece hash; context canceled; EOF Dec 18 14:26:48.541055401 (310h57m48.412138436s) [noise=true , address=77.90.31.11:4556 , piece_id=4GEBIFAQJYJX4KTPHTYUODQZ6ABTTPOA5SYOJBGUV7YDFIHHVO2Q, seg_pos={0 0}, upload=272678]: Long tail cancellation due to optimal threshold Dec 18 14:26:48.541086550 (310h57m48.412169595s) [seg_pos={0 0}, upload=272678]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 18 14:26:48.541155190 (310h57m48.412238226s) [upload=272678]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 18 14:26:48.541159900 (310h57m48.412242935s) [upload=272678]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 18 14:26:48.836195919 (310h57m48.707278954s) [upload=272678]: Done uploading segments... Dec 18 14:26:48.836215697 (310h57m48.707298742s) [upload=272678]: Upload finished. err: <nil> Duration: 5.82474206s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 21 out of 60 is failed (threshold=6) [X.X..............X.......X.....X.X.XX...XXXXX.XX..XX.XXXX...] duration: 3.497314964s trace: https://jaeger.dev.storj.io/trace/9dbfc1dffabb5ea FAIL (2025-12-18 14:36:29 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.497349039s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 21 out of 60 is failed (threshold=6) [.X.X..............X.......X.....X.X.XX...XXXXX.XX..XX.XXXX..] duration: 3.627937739s trace: https://jaeger.dev.storj.io/trace/9dbfc1dffabb5ea FAIL (2025-12-18 14:35:44 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.627967057s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........................X....................X...........] duration: 5.892158758s trace: https://jaeger.dev.storj.io/trace/2cbc07c217c30b3f FAIL (2025-12-18 14:35:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, Piece ID: O7IY6V6T4UILG3SXQ6K3SPXBVA2VLB22PWVQYIYORGLLYN6LE64A) context deadline exceeded; piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: I2SLWBEKFHZXWYTBZF73DICAZ4ZBXQYU6DES2QWLJMUCFZOCC7CA) context deadline exceeded ======LOGS====== ec 18 14:26:45.953557484 (310h57m45.824640529s) [upload=272678]: Starting upload Dec 18 14:26:45.953631205 (310h57m45.82471425s) [upload=272678]: Done starting upload Dec 18 14:26:45.953727147 (310h57m45.824810202s) [upload=272678]: Uploading segments... Dec 18 14:26:45.953920536 (310h57m45.825003581s) [upload=272678]: Got next segment. Inline: false Dec 18 14:26:45.953955743 (310h57m45.825038798s) [upload=272678]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 18 14:26:45.953963277 (310h57m45.825046322s) [upload=272678]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 18 14:26:46.201381593 (310h57m46.072464628s) [seg_pos={0 0}, upload=272678]: Begin upload segment... Dec 18 14:26:46.201430356 (310h57m46.072513391s) [seg_pos={0 0}, upload=272678]: Done begin upload segment. Dec 18 14:26:46.201523023 (310h57m46.072606088s) [noise=true , address=145.79.227.45:4231 , piece_id=V7B6RPI74SKRJQZCUWFHB2ECD46T3V224VKBL47NIIK5YH6PJCUQ, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.201534584 (310h57m46.072617619s) [noise=true , address=45.91.116.8:4048 , piece_id=BHQYLAJFKTBQ2WGY4XGRI4UOPTGB3SFEUVYZ7XWAC2BKA3ISU3WA, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.203237186 (310h57m46.074320231s) [noise=true , address=45.91.116.14:4190 , piece_id=I2SLWBEKFHZXWYTBZF73DICAZ4ZBXQYU6DES2QWLJMUCFZOCC7CA, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.203669961 (310h57m46.074752996s) [noise=true , address=145.79.227.36:4519 , piece_id=3E54JIZJCR55H36BEOLTDKWRMYA3WHNDPBSALSMNRY45O3BYO6JA, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.203970172 (310h57m46.075053207s) [noise=true , address=145.79.227.38:4978 , piece_id=FRWJ6Z3EH45YRMQVU235FJB75JWK7HOCLZXS7ASITOHLTL3YY5PQ, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.204250205 (310h57m46.07533324s) [noise=true , address=45.91.116.10:4084 , piece_id=UOTQVSUYETMA7UOEURUAT5RPX4QUO4FOXSLQY7UDUKJBCYCOH5BA, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.204534577 (310h57m46.075617612s) [noise=true , address=45.91.116.5:4299 , piece_id=SIZBVTWHVPXBL7OMRRQ467KFLRB56JNA5WDRNSXSNHPSV7UYG5LQ, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.204821995 (310h57m46.07590504s) [noise=true , address=145.79.227.29:4943 , piece_id=4DWWCYWOAU7NSESXDCONC6THXO3TTUYJMN7G6PTG2CP2OOOY2USA, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.206113864 (310h57m46.0771969s) [noise=true , address=77.90.31.11:4556 , piece_id=4GEBIFAQJYJX4KTPHTYUODQZ6ABTTPOA5SYOJBGUV7YDFIHHVO2Q, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.206151877 (310h57m46.077234912s) [noise=true , address=145.79.227.3:4150 , piece_id=XR5E35XW6KBFTREM2GFHARKAAIAINOWMLVYDKKBPFQBXJZOFCC2Q, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.207752204 (310h57m46.078835239s) [noise=true , address=145.79.227.39:4013 , piece_id=2SYWMQU5R7MCTH2E4X732AOVIRVPGUNNXLIN3UFCU7WF4BEJUACQ, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.210366552 (310h57m46.081449597s) [noise=true , address=145.79.227.35:4482 , piece_id=3EGA2CLZBLJWSSULQCBLEEHCMCJDE2FSOY4FCANAQXPEYTWIF53A, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.211613657 (310h57m46.082696692s) [noise=true , address=145.79.227.47:4266 , piece_id=4444BNJTIBUQWLNGTEK3IFD4TIWTCIBCWUHSYBMD3FGZSNDPWT3Q, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.212828561 (310h57m46.083911596s) [noise=true , address=145.79.227.41:4086 , piece_id=DFSNBYWUJ6Z65VTU5ME2IGEDB6KLMTAJM65RC44FUVYHTPEW4ZSA, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.215707031 (310h57m46.086790066s) [noise=true , address=45.91.116.13:4155 , piece_id=O7IY6V6T4UILG3SXQ6K3SPXBVA2VLB22PWVQYIYORGLLYN6LE64A, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.216889753 (310h57m46.087972798s) [noise=true , address=145.79.227.31:4797 , piece_id=F5WHLSRZSSZ4CRJRR2ZPYGGG67SHQ3QMAIHIC42NOVDW2UXARINQ, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:47.912125819 (310h57m47.783208865s) [upload=272678]: Next returned nil segment Dec 18 14:26:47.912143884 (310h57m47.783226929s) [upload=272678]: Waiting for error group managing segments... Dec 18 14:26:48.193212096 (310h57m48.064295131s) [noise=true , address=145.79.227.45:4231 , piece_id=V7B6RPI74SKRJQZCUWFHB2ECD46T3V224VKBL47NIIK5YH6PJCUQ, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.207387668 (310h57m48.078470703s) [noise=true , address=145.79.227.31:4797 , piece_id=F5WHLSRZSSZ4CRJRR2ZPYGGG67SHQ3QMAIHIC42NOVDW2UXARINQ, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.219064714 (310h57m48.090147759s) [noise=true , address=145.79.227.39:4013 , piece_id=2SYWMQU5R7MCTH2E4X732AOVIRVPGUNNXLIN3UFCU7WF4BEJUACQ, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.227887805 (310h57m48.09897085s) [noise=true , address=145.79.227.36:4519 , piece_id=3E54JIZJCR55H36BEOLTDKWRMYA3WHNDPBSALSMNRY45O3BYO6JA, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.229044407 (310h57m48.100127442s) [noise=true , address=145.79.227.41:4086 , piece_id=DFSNBYWUJ6Z65VTU5ME2IGEDB6KLMTAJM65RC44FUVYHTPEW4ZSA, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.234560812 (310h57m48.105643857s) [noise=true , address=145.79.227.47:4266 , piece_id=4444BNJTIBUQWLNGTEK3IFD4TIWTCIBCWUHSYBMD3FGZSNDPWT3Q, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.237048037 (310h57m48.108131072s) [noise=true , address=145.79.227.35:4482 , piece_id=3EGA2CLZBLJWSSULQCBLEEHCMCJDE2FSOY4FCANAQXPEYTWIF53A, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.267241984 (310h57m48.138325019s) [noise=true , address=145.79.227.29:4943 , piece_id=4DWWCYWOAU7NSESXDCONC6THXO3TTUYJMN7G6PTG2CP2OOOY2USA, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.288475576 (310h57m48.159558621s) [noise=true , address=145.79.227.3:4150 , piece_id=XR5E35XW6KBFTREM2GFHARKAAIAINOWMLVYDKKBPFQBXJZOFCC2Q, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.293478393 (310h57m48.164561458s) [noise=true , address=145.79.227.38:4978 , piece_id=FRWJ6Z3EH45YRMQVU235FJB75JWK7HOCLZXS7ASITOHLTL3YY5PQ, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.423759785 (310h57m48.29484282s) [noise=true , address=45.91.116.14:4190 , piece_id=I2SLWBEKFHZXWYTBZF73DICAZ4ZBXQYU6DES2QWLJMUCFZOCC7CA, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.485697347 (310h57m48.356780392s) [noise=true , address=45.91.116.10:4084 , piece_id=UOTQVSUYETMA7UOEURUAT5RPX4QUO4FOXSLQY7UDUKJBCYCOH5BA, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.540540711 (310h57m48.411623756s) [noise=true , address=45.91.116.13:4155 , piece_id=O7IY6V6T4UILG3SXQ6K3SPXBVA2VLB22PWVQYIYORGLLYN6LE64A, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.540575787 (310h57m48.411658832s) [seg_pos={0 0}, upload=272678]: Segment reached optimal threshold of 13 pieces. Dec 18 14:26:48.540831424 (310h57m48.411914459s) [noise=true , address=45.91.116.8:4048 , piece_id=BHQYLAJFKTBQ2WGY4XGRI4UOPTGB3SFEUVYZ7XWAC2BKA3ISU3WA, seg_pos={0 0}, upload=272678]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: write tcp 188.166.225.170:49328->45.91.116.8:4048: use of closed network connection; write tcp 188.166.225.170:49328->45.91.116.8:4048: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49328->45.91.116.8:4048: use of closed network connection Dec 18 14:26:48.540840511 (310h57m48.411923556s) [noise=true , address=45.91.116.8:4048 , piece_id=BHQYLAJFKTBQ2WGY4XGRI4UOPTGB3SFEUVYZ7XWAC2BKA3ISU3WA, seg_pos={0 0}, upload=272678]: Long tail cancellation due to optimal threshold Dec 18 14:26:48.540888052 (310h57m48.411971107s) [noise=true , address=45.91.116.5:4299 , piece_id=SIZBVTWHVPXBL7OMRRQ467KFLRB56JNA5WDRNSXSNHPSV7UYG5LQ, seg_pos={0 0}, upload=272678]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Dec 18 14:26:48.540899453 (310h57m48.411982498s) [noise=true , address=45.91.116.5:4299 , piece_id=SIZBVTWHVPXBL7OMRRQ467KFLRB56JNA5WDRNSXSNHPSV7UYG5LQ, seg_pos={0 0}, upload=272678]: Long tail cancellation due to optimal threshold Dec 18 14:26:48.541046764 (310h57m48.412129809s) [noise=true , address=77.90.31.11:4556 , piece_id=4GEBIFAQJYJX4KTPHTYUODQZ6ABTTPOA5SYOJBGUV7YDFIHHVO2Q, seg_pos={0 0}, upload=272678]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: expected piece hash; context canceled; EOF Dec 18 14:26:48.541055401 (310h57m48.412138436s) [noise=true , address=77.90.31.11:4556 , piece_id=4GEBIFAQJYJX4KTPHTYUODQZ6ABTTPOA5SYOJBGUV7YDFIHHVO2Q, seg_pos={0 0}, upload=272678]: Long tail cancellation due to optimal threshold Dec 18 14:26:48.541086550 (310h57m48.412169595s) [seg_pos={0 0}, upload=272678]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 18 14:26:48.541155190 (310h57m48.412238226s) [upload=272678]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 18 14:26:48.541159900 (310h57m48.412242935s) [upload=272678]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 18 14:26:48.836195919 (310h57m48.707278954s) [upload=272678]: Done uploading segments... Dec 18 14:26:48.836215697 (310h57m48.707298742s) [upload=272678]: Upload finished. err: <nil> Duration: 5.892188986s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 21 out of 60 is failed (threshold=6) [..X.X..............X.......X.....X.X.XX...XXXXX.XX..XX.XXXX.] duration: 5.00164653s trace: https://jaeger.dev.storj.io/trace/9dbfc1dffabb5ea FAIL (2025-12-18 14:34:51 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001679052s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [............................X....................X..........] duration: 5.681834356s trace: https://jaeger.dev.storj.io/trace/2cbc07c217c30b3f FAIL (2025-12-18 14:34:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, Piece ID: O7IY6V6T4UILG3SXQ6K3SPXBVA2VLB22PWVQYIYORGLLYN6LE64A) context deadline exceeded; piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: I2SLWBEKFHZXWYTBZF73DICAZ4ZBXQYU6DES2QWLJMUCFZOCC7CA) context deadline exceeded ======LOGS====== ec 18 14:26:45.953557484 (310h57m45.824640529s) [upload=272678]: Starting upload Dec 18 14:26:45.953631205 (310h57m45.82471425s) [upload=272678]: Done starting upload Dec 18 14:26:45.953727147 (310h57m45.824810202s) [upload=272678]: Uploading segments... Dec 18 14:26:45.953920536 (310h57m45.825003581s) [upload=272678]: Got next segment. Inline: false Dec 18 14:26:45.953955743 (310h57m45.825038798s) [upload=272678]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 18 14:26:45.953963277 (310h57m45.825046322s) [upload=272678]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 18 14:26:46.201381593 (310h57m46.072464628s) [seg_pos={0 0}, upload=272678]: Begin upload segment... Dec 18 14:26:46.201430356 (310h57m46.072513391s) [seg_pos={0 0}, upload=272678]: Done begin upload segment. Dec 18 14:26:46.201523023 (310h57m46.072606088s) [noise=true , address=145.79.227.45:4231 , piece_id=V7B6RPI74SKRJQZCUWFHB2ECD46T3V224VKBL47NIIK5YH6PJCUQ, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.201534584 (310h57m46.072617619s) [noise=true , address=45.91.116.8:4048 , piece_id=BHQYLAJFKTBQ2WGY4XGRI4UOPTGB3SFEUVYZ7XWAC2BKA3ISU3WA, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.203237186 (310h57m46.074320231s) [noise=true , address=45.91.116.14:4190 , piece_id=I2SLWBEKFHZXWYTBZF73DICAZ4ZBXQYU6DES2QWLJMUCFZOCC7CA, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.203669961 (310h57m46.074752996s) [noise=true , address=145.79.227.36:4519 , piece_id=3E54JIZJCR55H36BEOLTDKWRMYA3WHNDPBSALSMNRY45O3BYO6JA, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.203970172 (310h57m46.075053207s) [noise=true , address=145.79.227.38:4978 , piece_id=FRWJ6Z3EH45YRMQVU235FJB75JWK7HOCLZXS7ASITOHLTL3YY5PQ, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.204250205 (310h57m46.07533324s) [noise=true , address=45.91.116.10:4084 , piece_id=UOTQVSUYETMA7UOEURUAT5RPX4QUO4FOXSLQY7UDUKJBCYCOH5BA, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.204534577 (310h57m46.075617612s) [noise=true , address=45.91.116.5:4299 , piece_id=SIZBVTWHVPXBL7OMRRQ467KFLRB56JNA5WDRNSXSNHPSV7UYG5LQ, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.204821995 (310h57m46.07590504s) [noise=true , address=145.79.227.29:4943 , piece_id=4DWWCYWOAU7NSESXDCONC6THXO3TTUYJMN7G6PTG2CP2OOOY2USA, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.206113864 (310h57m46.0771969s) [noise=true , address=77.90.31.11:4556 , piece_id=4GEBIFAQJYJX4KTPHTYUODQZ6ABTTPOA5SYOJBGUV7YDFIHHVO2Q, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.206151877 (310h57m46.077234912s) [noise=true , address=145.79.227.3:4150 , piece_id=XR5E35XW6KBFTREM2GFHARKAAIAINOWMLVYDKKBPFQBXJZOFCC2Q, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.207752204 (310h57m46.078835239s) [noise=true , address=145.79.227.39:4013 , piece_id=2SYWMQU5R7MCTH2E4X732AOVIRVPGUNNXLIN3UFCU7WF4BEJUACQ, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.210366552 (310h57m46.081449597s) [noise=true , address=145.79.227.35:4482 , piece_id=3EGA2CLZBLJWSSULQCBLEEHCMCJDE2FSOY4FCANAQXPEYTWIF53A, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.211613657 (310h57m46.082696692s) [noise=true , address=145.79.227.47:4266 , piece_id=4444BNJTIBUQWLNGTEK3IFD4TIWTCIBCWUHSYBMD3FGZSNDPWT3Q, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.212828561 (310h57m46.083911596s) [noise=true , address=145.79.227.41:4086 , piece_id=DFSNBYWUJ6Z65VTU5ME2IGEDB6KLMTAJM65RC44FUVYHTPEW4ZSA, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.215707031 (310h57m46.086790066s) [noise=true , address=45.91.116.13:4155 , piece_id=O7IY6V6T4UILG3SXQ6K3SPXBVA2VLB22PWVQYIYORGLLYN6LE64A, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.216889753 (310h57m46.087972798s) [noise=true , address=145.79.227.31:4797 , piece_id=F5WHLSRZSSZ4CRJRR2ZPYGGG67SHQ3QMAIHIC42NOVDW2UXARINQ, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:47.912125819 (310h57m47.783208865s) [upload=272678]: Next returned nil segment Dec 18 14:26:47.912143884 (310h57m47.783226929s) [upload=272678]: Waiting for error group managing segments... Dec 18 14:26:48.193212096 (310h57m48.064295131s) [noise=true , address=145.79.227.45:4231 , piece_id=V7B6RPI74SKRJQZCUWFHB2ECD46T3V224VKBL47NIIK5YH6PJCUQ, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.207387668 (310h57m48.078470703s) [noise=true , address=145.79.227.31:4797 , piece_id=F5WHLSRZSSZ4CRJRR2ZPYGGG67SHQ3QMAIHIC42NOVDW2UXARINQ, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.219064714 (310h57m48.090147759s) [noise=true , address=145.79.227.39:4013 , piece_id=2SYWMQU5R7MCTH2E4X732AOVIRVPGUNNXLIN3UFCU7WF4BEJUACQ, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.227887805 (310h57m48.09897085s) [noise=true , address=145.79.227.36:4519 , piece_id=3E54JIZJCR55H36BEOLTDKWRMYA3WHNDPBSALSMNRY45O3BYO6JA, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.229044407 (310h57m48.100127442s) [noise=true , address=145.79.227.41:4086 , piece_id=DFSNBYWUJ6Z65VTU5ME2IGEDB6KLMTAJM65RC44FUVYHTPEW4ZSA, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.234560812 (310h57m48.105643857s) [noise=true , address=145.79.227.47:4266 , piece_id=4444BNJTIBUQWLNGTEK3IFD4TIWTCIBCWUHSYBMD3FGZSNDPWT3Q, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.237048037 (310h57m48.108131072s) [noise=true , address=145.79.227.35:4482 , piece_id=3EGA2CLZBLJWSSULQCBLEEHCMCJDE2FSOY4FCANAQXPEYTWIF53A, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.267241984 (310h57m48.138325019s) [noise=true , address=145.79.227.29:4943 , piece_id=4DWWCYWOAU7NSESXDCONC6THXO3TTUYJMN7G6PTG2CP2OOOY2USA, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.288475576 (310h57m48.159558621s) [noise=true , address=145.79.227.3:4150 , piece_id=XR5E35XW6KBFTREM2GFHARKAAIAINOWMLVYDKKBPFQBXJZOFCC2Q, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.293478393 (310h57m48.164561458s) [noise=true , address=145.79.227.38:4978 , piece_id=FRWJ6Z3EH45YRMQVU235FJB75JWK7HOCLZXS7ASITOHLTL3YY5PQ, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.423759785 (310h57m48.29484282s) [noise=true , address=45.91.116.14:4190 , piece_id=I2SLWBEKFHZXWYTBZF73DICAZ4ZBXQYU6DES2QWLJMUCFZOCC7CA, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.485697347 (310h57m48.356780392s) [noise=true , address=45.91.116.10:4084 , piece_id=UOTQVSUYETMA7UOEURUAT5RPX4QUO4FOXSLQY7UDUKJBCYCOH5BA, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.540540711 (310h57m48.411623756s) [noise=true , address=45.91.116.13:4155 , piece_id=O7IY6V6T4UILG3SXQ6K3SPXBVA2VLB22PWVQYIYORGLLYN6LE64A, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.540575787 (310h57m48.411658832s) [seg_pos={0 0}, upload=272678]: Segment reached optimal threshold of 13 pieces. Dec 18 14:26:48.540831424 (310h57m48.411914459s) [noise=true , address=45.91.116.8:4048 , piece_id=BHQYLAJFKTBQ2WGY4XGRI4UOPTGB3SFEUVYZ7XWAC2BKA3ISU3WA, seg_pos={0 0}, upload=272678]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: write tcp 188.166.225.170:49328->45.91.116.8:4048: use of closed network connection; write tcp 188.166.225.170:49328->45.91.116.8:4048: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49328->45.91.116.8:4048: use of closed network connection Dec 18 14:26:48.540840511 (310h57m48.411923556s) [noise=true , address=45.91.116.8:4048 , piece_id=BHQYLAJFKTBQ2WGY4XGRI4UOPTGB3SFEUVYZ7XWAC2BKA3ISU3WA, seg_pos={0 0}, upload=272678]: Long tail cancellation due to optimal threshold Dec 18 14:26:48.540888052 (310h57m48.411971107s) [noise=true , address=45.91.116.5:4299 , piece_id=SIZBVTWHVPXBL7OMRRQ467KFLRB56JNA5WDRNSXSNHPSV7UYG5LQ, seg_pos={0 0}, upload=272678]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Dec 18 14:26:48.540899453 (310h57m48.411982498s) [noise=true , address=45.91.116.5:4299 , piece_id=SIZBVTWHVPXBL7OMRRQ467KFLRB56JNA5WDRNSXSNHPSV7UYG5LQ, seg_pos={0 0}, upload=272678]: Long tail cancellation due to optimal threshold Dec 18 14:26:48.541046764 (310h57m48.412129809s) [noise=true , address=77.90.31.11:4556 , piece_id=4GEBIFAQJYJX4KTPHTYUODQZ6ABTTPOA5SYOJBGUV7YDFIHHVO2Q, seg_pos={0 0}, upload=272678]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: expected piece hash; context canceled; EOF Dec 18 14:26:48.541055401 (310h57m48.412138436s) [noise=true , address=77.90.31.11:4556 , piece_id=4GEBIFAQJYJX4KTPHTYUODQZ6ABTTPOA5SYOJBGUV7YDFIHHVO2Q, seg_pos={0 0}, upload=272678]: Long tail cancellation due to optimal threshold Dec 18 14:26:48.541086550 (310h57m48.412169595s) [seg_pos={0 0}, upload=272678]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 18 14:26:48.541155190 (310h57m48.412238226s) [upload=272678]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 18 14:26:48.541159900 (310h57m48.412242935s) [upload=272678]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 18 14:26:48.836195919 (310h57m48.707278954s) [upload=272678]: Done uploading segments... Dec 18 14:26:48.836215697 (310h57m48.707298742s) [upload=272678]: Upload finished. err: <nil> Duration: 5.681863381s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............................X....................X.........] duration: 5.797128682s trace: https://jaeger.dev.storj.io/trace/2cbc07c217c30b3f FAIL (2025-12-18 14:33:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, Piece ID: O7IY6V6T4UILG3SXQ6K3SPXBVA2VLB22PWVQYIYORGLLYN6LE64A) context deadline exceeded; piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: I2SLWBEKFHZXWYTBZF73DICAZ4ZBXQYU6DES2QWLJMUCFZOCC7CA) context deadline exceeded ======LOGS====== ec 18 14:26:45.953557484 (310h57m45.824640529s) [upload=272678]: Starting upload Dec 18 14:26:45.953631205 (310h57m45.82471425s) [upload=272678]: Done starting upload Dec 18 14:26:45.953727147 (310h57m45.824810202s) [upload=272678]: Uploading segments... Dec 18 14:26:45.953920536 (310h57m45.825003581s) [upload=272678]: Got next segment. Inline: false Dec 18 14:26:45.953955743 (310h57m45.825038798s) [upload=272678]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 18 14:26:45.953963277 (310h57m45.825046322s) [upload=272678]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 18 14:26:46.201381593 (310h57m46.072464628s) [seg_pos={0 0}, upload=272678]: Begin upload segment... Dec 18 14:26:46.201430356 (310h57m46.072513391s) [seg_pos={0 0}, upload=272678]: Done begin upload segment. Dec 18 14:26:46.201523023 (310h57m46.072606088s) [noise=true , address=145.79.227.45:4231 , piece_id=V7B6RPI74SKRJQZCUWFHB2ECD46T3V224VKBL47NIIK5YH6PJCUQ, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.201534584 (310h57m46.072617619s) [noise=true , address=45.91.116.8:4048 , piece_id=BHQYLAJFKTBQ2WGY4XGRI4UOPTGB3SFEUVYZ7XWAC2BKA3ISU3WA, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.203237186 (310h57m46.074320231s) [noise=true , address=45.91.116.14:4190 , piece_id=I2SLWBEKFHZXWYTBZF73DICAZ4ZBXQYU6DES2QWLJMUCFZOCC7CA, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.203669961 (310h57m46.074752996s) [noise=true , address=145.79.227.36:4519 , piece_id=3E54JIZJCR55H36BEOLTDKWRMYA3WHNDPBSALSMNRY45O3BYO6JA, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.203970172 (310h57m46.075053207s) [noise=true , address=145.79.227.38:4978 , piece_id=FRWJ6Z3EH45YRMQVU235FJB75JWK7HOCLZXS7ASITOHLTL3YY5PQ, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.204250205 (310h57m46.07533324s) [noise=true , address=45.91.116.10:4084 , piece_id=UOTQVSUYETMA7UOEURUAT5RPX4QUO4FOXSLQY7UDUKJBCYCOH5BA, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.204534577 (310h57m46.075617612s) [noise=true , address=45.91.116.5:4299 , piece_id=SIZBVTWHVPXBL7OMRRQ467KFLRB56JNA5WDRNSXSNHPSV7UYG5LQ, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.204821995 (310h57m46.07590504s) [noise=true , address=145.79.227.29:4943 , piece_id=4DWWCYWOAU7NSESXDCONC6THXO3TTUYJMN7G6PTG2CP2OOOY2USA, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.206113864 (310h57m46.0771969s) [noise=true , address=77.90.31.11:4556 , piece_id=4GEBIFAQJYJX4KTPHTYUODQZ6ABTTPOA5SYOJBGUV7YDFIHHVO2Q, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.206151877 (310h57m46.077234912s) [noise=true , address=145.79.227.3:4150 , piece_id=XR5E35XW6KBFTREM2GFHARKAAIAINOWMLVYDKKBPFQBXJZOFCC2Q, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.207752204 (310h57m46.078835239s) [noise=true , address=145.79.227.39:4013 , piece_id=2SYWMQU5R7MCTH2E4X732AOVIRVPGUNNXLIN3UFCU7WF4BEJUACQ, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.210366552 (310h57m46.081449597s) [noise=true , address=145.79.227.35:4482 , piece_id=3EGA2CLZBLJWSSULQCBLEEHCMCJDE2FSOY4FCANAQXPEYTWIF53A, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.211613657 (310h57m46.082696692s) [noise=true , address=145.79.227.47:4266 , piece_id=4444BNJTIBUQWLNGTEK3IFD4TIWTCIBCWUHSYBMD3FGZSNDPWT3Q, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.212828561 (310h57m46.083911596s) [noise=true , address=145.79.227.41:4086 , piece_id=DFSNBYWUJ6Z65VTU5ME2IGEDB6KLMTAJM65RC44FUVYHTPEW4ZSA, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.215707031 (310h57m46.086790066s) [noise=true , address=45.91.116.13:4155 , piece_id=O7IY6V6T4UILG3SXQ6K3SPXBVA2VLB22PWVQYIYORGLLYN6LE64A, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.216889753 (310h57m46.087972798s) [noise=true , address=145.79.227.31:4797 , piece_id=F5WHLSRZSSZ4CRJRR2ZPYGGG67SHQ3QMAIHIC42NOVDW2UXARINQ, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:47.912125819 (310h57m47.783208865s) [upload=272678]: Next returned nil segment Dec 18 14:26:47.912143884 (310h57m47.783226929s) [upload=272678]: Waiting for error group managing segments... Dec 18 14:26:48.193212096 (310h57m48.064295131s) [noise=true , address=145.79.227.45:4231 , piece_id=V7B6RPI74SKRJQZCUWFHB2ECD46T3V224VKBL47NIIK5YH6PJCUQ, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.207387668 (310h57m48.078470703s) [noise=true , address=145.79.227.31:4797 , piece_id=F5WHLSRZSSZ4CRJRR2ZPYGGG67SHQ3QMAIHIC42NOVDW2UXARINQ, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.219064714 (310h57m48.090147759s) [noise=true , address=145.79.227.39:4013 , piece_id=2SYWMQU5R7MCTH2E4X732AOVIRVPGUNNXLIN3UFCU7WF4BEJUACQ, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.227887805 (310h57m48.09897085s) [noise=true , address=145.79.227.36:4519 , piece_id=3E54JIZJCR55H36BEOLTDKWRMYA3WHNDPBSALSMNRY45O3BYO6JA, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.229044407 (310h57m48.100127442s) [noise=true , address=145.79.227.41:4086 , piece_id=DFSNBYWUJ6Z65VTU5ME2IGEDB6KLMTAJM65RC44FUVYHTPEW4ZSA, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.234560812 (310h57m48.105643857s) [noise=true , address=145.79.227.47:4266 , piece_id=4444BNJTIBUQWLNGTEK3IFD4TIWTCIBCWUHSYBMD3FGZSNDPWT3Q, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.237048037 (310h57m48.108131072s) [noise=true , address=145.79.227.35:4482 , piece_id=3EGA2CLZBLJWSSULQCBLEEHCMCJDE2FSOY4FCANAQXPEYTWIF53A, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.267241984 (310h57m48.138325019s) [noise=true , address=145.79.227.29:4943 , piece_id=4DWWCYWOAU7NSESXDCONC6THXO3TTUYJMN7G6PTG2CP2OOOY2USA, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.288475576 (310h57m48.159558621s) [noise=true , address=145.79.227.3:4150 , piece_id=XR5E35XW6KBFTREM2GFHARKAAIAINOWMLVYDKKBPFQBXJZOFCC2Q, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.293478393 (310h57m48.164561458s) [noise=true , address=145.79.227.38:4978 , piece_id=FRWJ6Z3EH45YRMQVU235FJB75JWK7HOCLZXS7ASITOHLTL3YY5PQ, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.423759785 (310h57m48.29484282s) [noise=true , address=45.91.116.14:4190 , piece_id=I2SLWBEKFHZXWYTBZF73DICAZ4ZBXQYU6DES2QWLJMUCFZOCC7CA, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.485697347 (310h57m48.356780392s) [noise=true , address=45.91.116.10:4084 , piece_id=UOTQVSUYETMA7UOEURUAT5RPX4QUO4FOXSLQY7UDUKJBCYCOH5BA, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.540540711 (310h57m48.411623756s) [noise=true , address=45.91.116.13:4155 , piece_id=O7IY6V6T4UILG3SXQ6K3SPXBVA2VLB22PWVQYIYORGLLYN6LE64A, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.540575787 (310h57m48.411658832s) [seg_pos={0 0}, upload=272678]: Segment reached optimal threshold of 13 pieces. Dec 18 14:26:48.540831424 (310h57m48.411914459s) [noise=true , address=45.91.116.8:4048 , piece_id=BHQYLAJFKTBQ2WGY4XGRI4UOPTGB3SFEUVYZ7XWAC2BKA3ISU3WA, seg_pos={0 0}, upload=272678]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: write tcp 188.166.225.170:49328->45.91.116.8:4048: use of closed network connection; write tcp 188.166.225.170:49328->45.91.116.8:4048: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49328->45.91.116.8:4048: use of closed network connection Dec 18 14:26:48.540840511 (310h57m48.411923556s) [noise=true , address=45.91.116.8:4048 , piece_id=BHQYLAJFKTBQ2WGY4XGRI4UOPTGB3SFEUVYZ7XWAC2BKA3ISU3WA, seg_pos={0 0}, upload=272678]: Long tail cancellation due to optimal threshold Dec 18 14:26:48.540888052 (310h57m48.411971107s) [noise=true , address=45.91.116.5:4299 , piece_id=SIZBVTWHVPXBL7OMRRQ467KFLRB56JNA5WDRNSXSNHPSV7UYG5LQ, seg_pos={0 0}, upload=272678]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Dec 18 14:26:48.540899453 (310h57m48.411982498s) [noise=true , address=45.91.116.5:4299 , piece_id=SIZBVTWHVPXBL7OMRRQ467KFLRB56JNA5WDRNSXSNHPSV7UYG5LQ, seg_pos={0 0}, upload=272678]: Long tail cancellation due to optimal threshold Dec 18 14:26:48.541046764 (310h57m48.412129809s) [noise=true , address=77.90.31.11:4556 , piece_id=4GEBIFAQJYJX4KTPHTYUODQZ6ABTTPOA5SYOJBGUV7YDFIHHVO2Q, seg_pos={0 0}, upload=272678]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: expected piece hash; context canceled; EOF Dec 18 14:26:48.541055401 (310h57m48.412138436s) [noise=true , address=77.90.31.11:4556 , piece_id=4GEBIFAQJYJX4KTPHTYUODQZ6ABTTPOA5SYOJBGUV7YDFIHHVO2Q, seg_pos={0 0}, upload=272678]: Long tail cancellation due to optimal threshold Dec 18 14:26:48.541086550 (310h57m48.412169595s) [seg_pos={0 0}, upload=272678]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 18 14:26:48.541155190 (310h57m48.412238226s) [upload=272678]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 18 14:26:48.541159900 (310h57m48.412242935s) [upload=272678]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 18 14:26:48.836195919 (310h57m48.707278954s) [upload=272678]: Done uploading segments... Dec 18 14:26:48.836215697 (310h57m48.707298742s) [upload=272678]: Upload finished. err: <nil> Duration: 5.79715457s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 22 out of 60 is failed (threshold=6) [X..X.X..............X.......X.....X.X.XX...XXXXX.XX..XX.XXXX] duration: 5.001890002s trace: https://jaeger.dev.storj.io/trace/9dbfc1dffabb5ea FAIL (2025-12-18 14:33:48 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001916203s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 21 out of 60 is failed (threshold=6) [.X..X.X..............X.......X.....X.X.XX...XXXXX.XX..XX.XXX] duration: 5.001336297s trace: https://jaeger.dev.storj.io/trace/2345aee4b1ec5cd0 FAIL (2025-12-18 14:33:16 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001493126s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............................X....................X........] duration: 5.776908198s trace: https://jaeger.dev.storj.io/trace/2cbc07c217c30b3f FAIL (2025-12-18 14:33:05 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, Piece ID: O7IY6V6T4UILG3SXQ6K3SPXBVA2VLB22PWVQYIYORGLLYN6LE64A) context deadline exceeded; piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: I2SLWBEKFHZXWYTBZF73DICAZ4ZBXQYU6DES2QWLJMUCFZOCC7CA) context deadline exceeded ======LOGS====== ec 18 14:26:45.953557484 (310h57m45.824640529s) [upload=272678]: Starting upload Dec 18 14:26:45.953631205 (310h57m45.82471425s) [upload=272678]: Done starting upload Dec 18 14:26:45.953727147 (310h57m45.824810202s) [upload=272678]: Uploading segments... Dec 18 14:26:45.953920536 (310h57m45.825003581s) [upload=272678]: Got next segment. Inline: false Dec 18 14:26:45.953955743 (310h57m45.825038798s) [upload=272678]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 18 14:26:45.953963277 (310h57m45.825046322s) [upload=272678]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 18 14:26:46.201381593 (310h57m46.072464628s) [seg_pos={0 0}, upload=272678]: Begin upload segment... Dec 18 14:26:46.201430356 (310h57m46.072513391s) [seg_pos={0 0}, upload=272678]: Done begin upload segment. Dec 18 14:26:46.201523023 (310h57m46.072606088s) [noise=true , address=145.79.227.45:4231 , piece_id=V7B6RPI74SKRJQZCUWFHB2ECD46T3V224VKBL47NIIK5YH6PJCUQ, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.201534584 (310h57m46.072617619s) [noise=true , address=45.91.116.8:4048 , piece_id=BHQYLAJFKTBQ2WGY4XGRI4UOPTGB3SFEUVYZ7XWAC2BKA3ISU3WA, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.203237186 (310h57m46.074320231s) [noise=true , address=45.91.116.14:4190 , piece_id=I2SLWBEKFHZXWYTBZF73DICAZ4ZBXQYU6DES2QWLJMUCFZOCC7CA, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.203669961 (310h57m46.074752996s) [noise=true , address=145.79.227.36:4519 , piece_id=3E54JIZJCR55H36BEOLTDKWRMYA3WHNDPBSALSMNRY45O3BYO6JA, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.203970172 (310h57m46.075053207s) [noise=true , address=145.79.227.38:4978 , piece_id=FRWJ6Z3EH45YRMQVU235FJB75JWK7HOCLZXS7ASITOHLTL3YY5PQ, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.204250205 (310h57m46.07533324s) [noise=true , address=45.91.116.10:4084 , piece_id=UOTQVSUYETMA7UOEURUAT5RPX4QUO4FOXSLQY7UDUKJBCYCOH5BA, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.204534577 (310h57m46.075617612s) [noise=true , address=45.91.116.5:4299 , piece_id=SIZBVTWHVPXBL7OMRRQ467KFLRB56JNA5WDRNSXSNHPSV7UYG5LQ, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.204821995 (310h57m46.07590504s) [noise=true , address=145.79.227.29:4943 , piece_id=4DWWCYWOAU7NSESXDCONC6THXO3TTUYJMN7G6PTG2CP2OOOY2USA, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.206113864 (310h57m46.0771969s) [noise=true , address=77.90.31.11:4556 , piece_id=4GEBIFAQJYJX4KTPHTYUODQZ6ABTTPOA5SYOJBGUV7YDFIHHVO2Q, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.206151877 (310h57m46.077234912s) [noise=true , address=145.79.227.3:4150 , piece_id=XR5E35XW6KBFTREM2GFHARKAAIAINOWMLVYDKKBPFQBXJZOFCC2Q, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.207752204 (310h57m46.078835239s) [noise=true , address=145.79.227.39:4013 , piece_id=2SYWMQU5R7MCTH2E4X732AOVIRVPGUNNXLIN3UFCU7WF4BEJUACQ, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.210366552 (310h57m46.081449597s) [noise=true , address=145.79.227.35:4482 , piece_id=3EGA2CLZBLJWSSULQCBLEEHCMCJDE2FSOY4FCANAQXPEYTWIF53A, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.211613657 (310h57m46.082696692s) [noise=true , address=145.79.227.47:4266 , piece_id=4444BNJTIBUQWLNGTEK3IFD4TIWTCIBCWUHSYBMD3FGZSNDPWT3Q, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.212828561 (310h57m46.083911596s) [noise=true , address=145.79.227.41:4086 , piece_id=DFSNBYWUJ6Z65VTU5ME2IGEDB6KLMTAJM65RC44FUVYHTPEW4ZSA, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.215707031 (310h57m46.086790066s) [noise=true , address=45.91.116.13:4155 , piece_id=O7IY6V6T4UILG3SXQ6K3SPXBVA2VLB22PWVQYIYORGLLYN6LE64A, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.216889753 (310h57m46.087972798s) [noise=true , address=145.79.227.31:4797 , piece_id=F5WHLSRZSSZ4CRJRR2ZPYGGG67SHQ3QMAIHIC42NOVDW2UXARINQ, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:47.912125819 (310h57m47.783208865s) [upload=272678]: Next returned nil segment Dec 18 14:26:47.912143884 (310h57m47.783226929s) [upload=272678]: Waiting for error group managing segments... Dec 18 14:26:48.193212096 (310h57m48.064295131s) [noise=true , address=145.79.227.45:4231 , piece_id=V7B6RPI74SKRJQZCUWFHB2ECD46T3V224VKBL47NIIK5YH6PJCUQ, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.207387668 (310h57m48.078470703s) [noise=true , address=145.79.227.31:4797 , piece_id=F5WHLSRZSSZ4CRJRR2ZPYGGG67SHQ3QMAIHIC42NOVDW2UXARINQ, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.219064714 (310h57m48.090147759s) [noise=true , address=145.79.227.39:4013 , piece_id=2SYWMQU5R7MCTH2E4X732AOVIRVPGUNNXLIN3UFCU7WF4BEJUACQ, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.227887805 (310h57m48.09897085s) [noise=true , address=145.79.227.36:4519 , piece_id=3E54JIZJCR55H36BEOLTDKWRMYA3WHNDPBSALSMNRY45O3BYO6JA, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.229044407 (310h57m48.100127442s) [noise=true , address=145.79.227.41:4086 , piece_id=DFSNBYWUJ6Z65VTU5ME2IGEDB6KLMTAJM65RC44FUVYHTPEW4ZSA, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.234560812 (310h57m48.105643857s) [noise=true , address=145.79.227.47:4266 , piece_id=4444BNJTIBUQWLNGTEK3IFD4TIWTCIBCWUHSYBMD3FGZSNDPWT3Q, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.237048037 (310h57m48.108131072s) [noise=true , address=145.79.227.35:4482 , piece_id=3EGA2CLZBLJWSSULQCBLEEHCMCJDE2FSOY4FCANAQXPEYTWIF53A, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.267241984 (310h57m48.138325019s) [noise=true , address=145.79.227.29:4943 , piece_id=4DWWCYWOAU7NSESXDCONC6THXO3TTUYJMN7G6PTG2CP2OOOY2USA, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.288475576 (310h57m48.159558621s) [noise=true , address=145.79.227.3:4150 , piece_id=XR5E35XW6KBFTREM2GFHARKAAIAINOWMLVYDKKBPFQBXJZOFCC2Q, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.293478393 (310h57m48.164561458s) [noise=true , address=145.79.227.38:4978 , piece_id=FRWJ6Z3EH45YRMQVU235FJB75JWK7HOCLZXS7ASITOHLTL3YY5PQ, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.423759785 (310h57m48.29484282s) [noise=true , address=45.91.116.14:4190 , piece_id=I2SLWBEKFHZXWYTBZF73DICAZ4ZBXQYU6DES2QWLJMUCFZOCC7CA, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.485697347 (310h57m48.356780392s) [noise=true , address=45.91.116.10:4084 , piece_id=UOTQVSUYETMA7UOEURUAT5RPX4QUO4FOXSLQY7UDUKJBCYCOH5BA, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.540540711 (310h57m48.411623756s) [noise=true , address=45.91.116.13:4155 , piece_id=O7IY6V6T4UILG3SXQ6K3SPXBVA2VLB22PWVQYIYORGLLYN6LE64A, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.540575787 (310h57m48.411658832s) [seg_pos={0 0}, upload=272678]: Segment reached optimal threshold of 13 pieces. Dec 18 14:26:48.540831424 (310h57m48.411914459s) [noise=true , address=45.91.116.8:4048 , piece_id=BHQYLAJFKTBQ2WGY4XGRI4UOPTGB3SFEUVYZ7XWAC2BKA3ISU3WA, seg_pos={0 0}, upload=272678]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: write tcp 188.166.225.170:49328->45.91.116.8:4048: use of closed network connection; write tcp 188.166.225.170:49328->45.91.116.8:4048: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49328->45.91.116.8:4048: use of closed network connection Dec 18 14:26:48.540840511 (310h57m48.411923556s) [noise=true , address=45.91.116.8:4048 , piece_id=BHQYLAJFKTBQ2WGY4XGRI4UOPTGB3SFEUVYZ7XWAC2BKA3ISU3WA, seg_pos={0 0}, upload=272678]: Long tail cancellation due to optimal threshold Dec 18 14:26:48.540888052 (310h57m48.411971107s) [noise=true , address=45.91.116.5:4299 , piece_id=SIZBVTWHVPXBL7OMRRQ467KFLRB56JNA5WDRNSXSNHPSV7UYG5LQ, seg_pos={0 0}, upload=272678]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Dec 18 14:26:48.540899453 (310h57m48.411982498s) [noise=true , address=45.91.116.5:4299 , piece_id=SIZBVTWHVPXBL7OMRRQ467KFLRB56JNA5WDRNSXSNHPSV7UYG5LQ, seg_pos={0 0}, upload=272678]: Long tail cancellation due to optimal threshold Dec 18 14:26:48.541046764 (310h57m48.412129809s) [noise=true , address=77.90.31.11:4556 , piece_id=4GEBIFAQJYJX4KTPHTYUODQZ6ABTTPOA5SYOJBGUV7YDFIHHVO2Q, seg_pos={0 0}, upload=272678]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: expected piece hash; context canceled; EOF Dec 18 14:26:48.541055401 (310h57m48.412138436s) [noise=true , address=77.90.31.11:4556 , piece_id=4GEBIFAQJYJX4KTPHTYUODQZ6ABTTPOA5SYOJBGUV7YDFIHHVO2Q, seg_pos={0 0}, upload=272678]: Long tail cancellation due to optimal threshold Dec 18 14:26:48.541086550 (310h57m48.412169595s) [seg_pos={0 0}, upload=272678]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 18 14:26:48.541155190 (310h57m48.412238226s) [upload=272678]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 18 14:26:48.541159900 (310h57m48.412242935s) [upload=272678]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 18 14:26:48.836195919 (310h57m48.707278954s) [upload=272678]: Done uploading segments... Dec 18 14:26:48.836215697 (310h57m48.707298742s) [upload=272678]: Upload finished. err: <nil> Duration: 5.776940288s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 20 out of 60 is failed (threshold=6) [..X..X.X..............X.......X.....X.X.XX...XXXXX.XX..XX.XX] duration: 5.0021612s trace: https://jaeger.dev.storj.io/trace/66eddf3214732455 FAIL (2025-12-18 14:32:45 UTC) orbiter warning: context deadline exceeded Duration: 5.00218662s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 19 out of 60 is failed (threshold=6) [...X..X.X..............X.......X.....X.X.XX...XXXXX.XX..XX.X] duration: 5.001108023s trace: https://jaeger.dev.storj.io/trace/7152737c2114eb32 FAIL (2025-12-18 14:32:13 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001132228s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............................X....................X.......] duration: 5.813602871s trace: https://jaeger.dev.storj.io/trace/2cbc07c217c30b3f FAIL (2025-12-18 14:32:00 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, Piece ID: O7IY6V6T4UILG3SXQ6K3SPXBVA2VLB22PWVQYIYORGLLYN6LE64A) context deadline exceeded; piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: I2SLWBEKFHZXWYTBZF73DICAZ4ZBXQYU6DES2QWLJMUCFZOCC7CA) context deadline exceeded ======LOGS====== ec 18 14:26:45.953557484 (310h57m45.824640529s) [upload=272678]: Starting upload Dec 18 14:26:45.953631205 (310h57m45.82471425s) [upload=272678]: Done starting upload Dec 18 14:26:45.953727147 (310h57m45.824810202s) [upload=272678]: Uploading segments... Dec 18 14:26:45.953920536 (310h57m45.825003581s) [upload=272678]: Got next segment. Inline: false Dec 18 14:26:45.953955743 (310h57m45.825038798s) [upload=272678]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 18 14:26:45.953963277 (310h57m45.825046322s) [upload=272678]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 18 14:26:46.201381593 (310h57m46.072464628s) [seg_pos={0 0}, upload=272678]: Begin upload segment... Dec 18 14:26:46.201430356 (310h57m46.072513391s) [seg_pos={0 0}, upload=272678]: Done begin upload segment. Dec 18 14:26:46.201523023 (310h57m46.072606088s) [noise=true , address=145.79.227.45:4231 , piece_id=V7B6RPI74SKRJQZCUWFHB2ECD46T3V224VKBL47NIIK5YH6PJCUQ, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.201534584 (310h57m46.072617619s) [noise=true , address=45.91.116.8:4048 , piece_id=BHQYLAJFKTBQ2WGY4XGRI4UOPTGB3SFEUVYZ7XWAC2BKA3ISU3WA, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.203237186 (310h57m46.074320231s) [noise=true , address=45.91.116.14:4190 , piece_id=I2SLWBEKFHZXWYTBZF73DICAZ4ZBXQYU6DES2QWLJMUCFZOCC7CA, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.203669961 (310h57m46.074752996s) [noise=true , address=145.79.227.36:4519 , piece_id=3E54JIZJCR55H36BEOLTDKWRMYA3WHNDPBSALSMNRY45O3BYO6JA, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.203970172 (310h57m46.075053207s) [noise=true , address=145.79.227.38:4978 , piece_id=FRWJ6Z3EH45YRMQVU235FJB75JWK7HOCLZXS7ASITOHLTL3YY5PQ, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.204250205 (310h57m46.07533324s) [noise=true , address=45.91.116.10:4084 , piece_id=UOTQVSUYETMA7UOEURUAT5RPX4QUO4FOXSLQY7UDUKJBCYCOH5BA, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.204534577 (310h57m46.075617612s) [noise=true , address=45.91.116.5:4299 , piece_id=SIZBVTWHVPXBL7OMRRQ467KFLRB56JNA5WDRNSXSNHPSV7UYG5LQ, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.204821995 (310h57m46.07590504s) [noise=true , address=145.79.227.29:4943 , piece_id=4DWWCYWOAU7NSESXDCONC6THXO3TTUYJMN7G6PTG2CP2OOOY2USA, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.206113864 (310h57m46.0771969s) [noise=true , address=77.90.31.11:4556 , piece_id=4GEBIFAQJYJX4KTPHTYUODQZ6ABTTPOA5SYOJBGUV7YDFIHHVO2Q, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.206151877 (310h57m46.077234912s) [noise=true , address=145.79.227.3:4150 , piece_id=XR5E35XW6KBFTREM2GFHARKAAIAINOWMLVYDKKBPFQBXJZOFCC2Q, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.207752204 (310h57m46.078835239s) [noise=true , address=145.79.227.39:4013 , piece_id=2SYWMQU5R7MCTH2E4X732AOVIRVPGUNNXLIN3UFCU7WF4BEJUACQ, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.210366552 (310h57m46.081449597s) [noise=true , address=145.79.227.35:4482 , piece_id=3EGA2CLZBLJWSSULQCBLEEHCMCJDE2FSOY4FCANAQXPEYTWIF53A, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.211613657 (310h57m46.082696692s) [noise=true , address=145.79.227.47:4266 , piece_id=4444BNJTIBUQWLNGTEK3IFD4TIWTCIBCWUHSYBMD3FGZSNDPWT3Q, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.212828561 (310h57m46.083911596s) [noise=true , address=145.79.227.41:4086 , piece_id=DFSNBYWUJ6Z65VTU5ME2IGEDB6KLMTAJM65RC44FUVYHTPEW4ZSA, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.215707031 (310h57m46.086790066s) [noise=true , address=45.91.116.13:4155 , piece_id=O7IY6V6T4UILG3SXQ6K3SPXBVA2VLB22PWVQYIYORGLLYN6LE64A, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.216889753 (310h57m46.087972798s) [noise=true , address=145.79.227.31:4797 , piece_id=F5WHLSRZSSZ4CRJRR2ZPYGGG67SHQ3QMAIHIC42NOVDW2UXARINQ, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:47.912125819 (310h57m47.783208865s) [upload=272678]: Next returned nil segment Dec 18 14:26:47.912143884 (310h57m47.783226929s) [upload=272678]: Waiting for error group managing segments... Dec 18 14:26:48.193212096 (310h57m48.064295131s) [noise=true , address=145.79.227.45:4231 , piece_id=V7B6RPI74SKRJQZCUWFHB2ECD46T3V224VKBL47NIIK5YH6PJCUQ, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.207387668 (310h57m48.078470703s) [noise=true , address=145.79.227.31:4797 , piece_id=F5WHLSRZSSZ4CRJRR2ZPYGGG67SHQ3QMAIHIC42NOVDW2UXARINQ, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.219064714 (310h57m48.090147759s) [noise=true , address=145.79.227.39:4013 , piece_id=2SYWMQU5R7MCTH2E4X732AOVIRVPGUNNXLIN3UFCU7WF4BEJUACQ, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.227887805 (310h57m48.09897085s) [noise=true , address=145.79.227.36:4519 , piece_id=3E54JIZJCR55H36BEOLTDKWRMYA3WHNDPBSALSMNRY45O3BYO6JA, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.229044407 (310h57m48.100127442s) [noise=true , address=145.79.227.41:4086 , piece_id=DFSNBYWUJ6Z65VTU5ME2IGEDB6KLMTAJM65RC44FUVYHTPEW4ZSA, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.234560812 (310h57m48.105643857s) [noise=true , address=145.79.227.47:4266 , piece_id=4444BNJTIBUQWLNGTEK3IFD4TIWTCIBCWUHSYBMD3FGZSNDPWT3Q, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.237048037 (310h57m48.108131072s) [noise=true , address=145.79.227.35:4482 , piece_id=3EGA2CLZBLJWSSULQCBLEEHCMCJDE2FSOY4FCANAQXPEYTWIF53A, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.267241984 (310h57m48.138325019s) [noise=true , address=145.79.227.29:4943 , piece_id=4DWWCYWOAU7NSESXDCONC6THXO3TTUYJMN7G6PTG2CP2OOOY2USA, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.288475576 (310h57m48.159558621s) [noise=true , address=145.79.227.3:4150 , piece_id=XR5E35XW6KBFTREM2GFHARKAAIAINOWMLVYDKKBPFQBXJZOFCC2Q, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.293478393 (310h57m48.164561458s) [noise=true , address=145.79.227.38:4978 , piece_id=FRWJ6Z3EH45YRMQVU235FJB75JWK7HOCLZXS7ASITOHLTL3YY5PQ, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.423759785 (310h57m48.29484282s) [noise=true , address=45.91.116.14:4190 , piece_id=I2SLWBEKFHZXWYTBZF73DICAZ4ZBXQYU6DES2QWLJMUCFZOCC7CA, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.485697347 (310h57m48.356780392s) [noise=true , address=45.91.116.10:4084 , piece_id=UOTQVSUYETMA7UOEURUAT5RPX4QUO4FOXSLQY7UDUKJBCYCOH5BA, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.540540711 (310h57m48.411623756s) [noise=true , address=45.91.116.13:4155 , piece_id=O7IY6V6T4UILG3SXQ6K3SPXBVA2VLB22PWVQYIYORGLLYN6LE64A, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.540575787 (310h57m48.411658832s) [seg_pos={0 0}, upload=272678]: Segment reached optimal threshold of 13 pieces. Dec 18 14:26:48.540831424 (310h57m48.411914459s) [noise=true , address=45.91.116.8:4048 , piece_id=BHQYLAJFKTBQ2WGY4XGRI4UOPTGB3SFEUVYZ7XWAC2BKA3ISU3WA, seg_pos={0 0}, upload=272678]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: write tcp 188.166.225.170:49328->45.91.116.8:4048: use of closed network connection; write tcp 188.166.225.170:49328->45.91.116.8:4048: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49328->45.91.116.8:4048: use of closed network connection Dec 18 14:26:48.540840511 (310h57m48.411923556s) [noise=true , address=45.91.116.8:4048 , piece_id=BHQYLAJFKTBQ2WGY4XGRI4UOPTGB3SFEUVYZ7XWAC2BKA3ISU3WA, seg_pos={0 0}, upload=272678]: Long tail cancellation due to optimal threshold Dec 18 14:26:48.540888052 (310h57m48.411971107s) [noise=true , address=45.91.116.5:4299 , piece_id=SIZBVTWHVPXBL7OMRRQ467KFLRB56JNA5WDRNSXSNHPSV7UYG5LQ, seg_pos={0 0}, upload=272678]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Dec 18 14:26:48.540899453 (310h57m48.411982498s) [noise=true , address=45.91.116.5:4299 , piece_id=SIZBVTWHVPXBL7OMRRQ467KFLRB56JNA5WDRNSXSNHPSV7UYG5LQ, seg_pos={0 0}, upload=272678]: Long tail cancellation due to optimal threshold Dec 18 14:26:48.541046764 (310h57m48.412129809s) [noise=true , address=77.90.31.11:4556 , piece_id=4GEBIFAQJYJX4KTPHTYUODQZ6ABTTPOA5SYOJBGUV7YDFIHHVO2Q, seg_pos={0 0}, upload=272678]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: expected piece hash; context canceled; EOF Dec 18 14:26:48.541055401 (310h57m48.412138436s) [noise=true , address=77.90.31.11:4556 , piece_id=4GEBIFAQJYJX4KTPHTYUODQZ6ABTTPOA5SYOJBGUV7YDFIHHVO2Q, seg_pos={0 0}, upload=272678]: Long tail cancellation due to optimal threshold Dec 18 14:26:48.541086550 (310h57m48.412169595s) [seg_pos={0 0}, upload=272678]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 18 14:26:48.541155190 (310h57m48.412238226s) [upload=272678]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 18 14:26:48.541159900 (310h57m48.412242935s) [upload=272678]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 18 14:26:48.836195919 (310h57m48.707278954s) [upload=272678]: Done uploading segments... Dec 18 14:26:48.836215697 (310h57m48.707298742s) [upload=272678]: Upload finished. err: <nil> Duration: 5.813636436s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [................................X....................X......] duration: 5.972799015s trace: https://jaeger.dev.storj.io/trace/2cbc07c217c30b3f FAIL (2025-12-18 14:31:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, Piece ID: O7IY6V6T4UILG3SXQ6K3SPXBVA2VLB22PWVQYIYORGLLYN6LE64A) context deadline exceeded; piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: I2SLWBEKFHZXWYTBZF73DICAZ4ZBXQYU6DES2QWLJMUCFZOCC7CA) context deadline exceeded ======LOGS====== ec 18 14:26:45.953557484 (310h57m45.824640529s) [upload=272678]: Starting upload Dec 18 14:26:45.953631205 (310h57m45.82471425s) [upload=272678]: Done starting upload Dec 18 14:26:45.953727147 (310h57m45.824810202s) [upload=272678]: Uploading segments... Dec 18 14:26:45.953920536 (310h57m45.825003581s) [upload=272678]: Got next segment. Inline: false Dec 18 14:26:45.953955743 (310h57m45.825038798s) [upload=272678]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 18 14:26:45.953963277 (310h57m45.825046322s) [upload=272678]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 18 14:26:46.201381593 (310h57m46.072464628s) [seg_pos={0 0}, upload=272678]: Begin upload segment... Dec 18 14:26:46.201430356 (310h57m46.072513391s) [seg_pos={0 0}, upload=272678]: Done begin upload segment. Dec 18 14:26:46.201523023 (310h57m46.072606088s) [noise=true , address=145.79.227.45:4231 , piece_id=V7B6RPI74SKRJQZCUWFHB2ECD46T3V224VKBL47NIIK5YH6PJCUQ, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.201534584 (310h57m46.072617619s) [noise=true , address=45.91.116.8:4048 , piece_id=BHQYLAJFKTBQ2WGY4XGRI4UOPTGB3SFEUVYZ7XWAC2BKA3ISU3WA, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.203237186 (310h57m46.074320231s) [noise=true , address=45.91.116.14:4190 , piece_id=I2SLWBEKFHZXWYTBZF73DICAZ4ZBXQYU6DES2QWLJMUCFZOCC7CA, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.203669961 (310h57m46.074752996s) [noise=true , address=145.79.227.36:4519 , piece_id=3E54JIZJCR55H36BEOLTDKWRMYA3WHNDPBSALSMNRY45O3BYO6JA, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.203970172 (310h57m46.075053207s) [noise=true , address=145.79.227.38:4978 , piece_id=FRWJ6Z3EH45YRMQVU235FJB75JWK7HOCLZXS7ASITOHLTL3YY5PQ, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.204250205 (310h57m46.07533324s) [noise=true , address=45.91.116.10:4084 , piece_id=UOTQVSUYETMA7UOEURUAT5RPX4QUO4FOXSLQY7UDUKJBCYCOH5BA, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.204534577 (310h57m46.075617612s) [noise=true , address=45.91.116.5:4299 , piece_id=SIZBVTWHVPXBL7OMRRQ467KFLRB56JNA5WDRNSXSNHPSV7UYG5LQ, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.204821995 (310h57m46.07590504s) [noise=true , address=145.79.227.29:4943 , piece_id=4DWWCYWOAU7NSESXDCONC6THXO3TTUYJMN7G6PTG2CP2OOOY2USA, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.206113864 (310h57m46.0771969s) [noise=true , address=77.90.31.11:4556 , piece_id=4GEBIFAQJYJX4KTPHTYUODQZ6ABTTPOA5SYOJBGUV7YDFIHHVO2Q, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.206151877 (310h57m46.077234912s) [noise=true , address=145.79.227.3:4150 , piece_id=XR5E35XW6KBFTREM2GFHARKAAIAINOWMLVYDKKBPFQBXJZOFCC2Q, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.207752204 (310h57m46.078835239s) [noise=true , address=145.79.227.39:4013 , piece_id=2SYWMQU5R7MCTH2E4X732AOVIRVPGUNNXLIN3UFCU7WF4BEJUACQ, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.210366552 (310h57m46.081449597s) [noise=true , address=145.79.227.35:4482 , piece_id=3EGA2CLZBLJWSSULQCBLEEHCMCJDE2FSOY4FCANAQXPEYTWIF53A, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.211613657 (310h57m46.082696692s) [noise=true , address=145.79.227.47:4266 , piece_id=4444BNJTIBUQWLNGTEK3IFD4TIWTCIBCWUHSYBMD3FGZSNDPWT3Q, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.212828561 (310h57m46.083911596s) [noise=true , address=145.79.227.41:4086 , piece_id=DFSNBYWUJ6Z65VTU5ME2IGEDB6KLMTAJM65RC44FUVYHTPEW4ZSA, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.215707031 (310h57m46.086790066s) [noise=true , address=45.91.116.13:4155 , piece_id=O7IY6V6T4UILG3SXQ6K3SPXBVA2VLB22PWVQYIYORGLLYN6LE64A, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.216889753 (310h57m46.087972798s) [noise=true , address=145.79.227.31:4797 , piece_id=F5WHLSRZSSZ4CRJRR2ZPYGGG67SHQ3QMAIHIC42NOVDW2UXARINQ, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:47.912125819 (310h57m47.783208865s) [upload=272678]: Next returned nil segment Dec 18 14:26:47.912143884 (310h57m47.783226929s) [upload=272678]: Waiting for error group managing segments... Dec 18 14:26:48.193212096 (310h57m48.064295131s) [noise=true , address=145.79.227.45:4231 , piece_id=V7B6RPI74SKRJQZCUWFHB2ECD46T3V224VKBL47NIIK5YH6PJCUQ, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.207387668 (310h57m48.078470703s) [noise=true , address=145.79.227.31:4797 , piece_id=F5WHLSRZSSZ4CRJRR2ZPYGGG67SHQ3QMAIHIC42NOVDW2UXARINQ, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.219064714 (310h57m48.090147759s) [noise=true , address=145.79.227.39:4013 , piece_id=2SYWMQU5R7MCTH2E4X732AOVIRVPGUNNXLIN3UFCU7WF4BEJUACQ, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.227887805 (310h57m48.09897085s) [noise=true , address=145.79.227.36:4519 , piece_id=3E54JIZJCR55H36BEOLTDKWRMYA3WHNDPBSALSMNRY45O3BYO6JA, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.229044407 (310h57m48.100127442s) [noise=true , address=145.79.227.41:4086 , piece_id=DFSNBYWUJ6Z65VTU5ME2IGEDB6KLMTAJM65RC44FUVYHTPEW4ZSA, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.234560812 (310h57m48.105643857s) [noise=true , address=145.79.227.47:4266 , piece_id=4444BNJTIBUQWLNGTEK3IFD4TIWTCIBCWUHSYBMD3FGZSNDPWT3Q, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.237048037 (310h57m48.108131072s) [noise=true , address=145.79.227.35:4482 , piece_id=3EGA2CLZBLJWSSULQCBLEEHCMCJDE2FSOY4FCANAQXPEYTWIF53A, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.267241984 (310h57m48.138325019s) [noise=true , address=145.79.227.29:4943 , piece_id=4DWWCYWOAU7NSESXDCONC6THXO3TTUYJMN7G6PTG2CP2OOOY2USA, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.288475576 (310h57m48.159558621s) [noise=true , address=145.79.227.3:4150 , piece_id=XR5E35XW6KBFTREM2GFHARKAAIAINOWMLVYDKKBPFQBXJZOFCC2Q, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.293478393 (310h57m48.164561458s) [noise=true , address=145.79.227.38:4978 , piece_id=FRWJ6Z3EH45YRMQVU235FJB75JWK7HOCLZXS7ASITOHLTL3YY5PQ, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.423759785 (310h57m48.29484282s) [noise=true , address=45.91.116.14:4190 , piece_id=I2SLWBEKFHZXWYTBZF73DICAZ4ZBXQYU6DES2QWLJMUCFZOCC7CA, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.485697347 (310h57m48.356780392s) [noise=true , address=45.91.116.10:4084 , piece_id=UOTQVSUYETMA7UOEURUAT5RPX4QUO4FOXSLQY7UDUKJBCYCOH5BA, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.540540711 (310h57m48.411623756s) [noise=true , address=45.91.116.13:4155 , piece_id=O7IY6V6T4UILG3SXQ6K3SPXBVA2VLB22PWVQYIYORGLLYN6LE64A, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.540575787 (310h57m48.411658832s) [seg_pos={0 0}, upload=272678]: Segment reached optimal threshold of 13 pieces. Dec 18 14:26:48.540831424 (310h57m48.411914459s) [noise=true , address=45.91.116.8:4048 , piece_id=BHQYLAJFKTBQ2WGY4XGRI4UOPTGB3SFEUVYZ7XWAC2BKA3ISU3WA, seg_pos={0 0}, upload=272678]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: write tcp 188.166.225.170:49328->45.91.116.8:4048: use of closed network connection; write tcp 188.166.225.170:49328->45.91.116.8:4048: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49328->45.91.116.8:4048: use of closed network connection Dec 18 14:26:48.540840511 (310h57m48.411923556s) [noise=true , address=45.91.116.8:4048 , piece_id=BHQYLAJFKTBQ2WGY4XGRI4UOPTGB3SFEUVYZ7XWAC2BKA3ISU3WA, seg_pos={0 0}, upload=272678]: Long tail cancellation due to optimal threshold Dec 18 14:26:48.540888052 (310h57m48.411971107s) [noise=true , address=45.91.116.5:4299 , piece_id=SIZBVTWHVPXBL7OMRRQ467KFLRB56JNA5WDRNSXSNHPSV7UYG5LQ, seg_pos={0 0}, upload=272678]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Dec 18 14:26:48.540899453 (310h57m48.411982498s) [noise=true , address=45.91.116.5:4299 , piece_id=SIZBVTWHVPXBL7OMRRQ467KFLRB56JNA5WDRNSXSNHPSV7UYG5LQ, seg_pos={0 0}, upload=272678]: Long tail cancellation due to optimal threshold Dec 18 14:26:48.541046764 (310h57m48.412129809s) [noise=true , address=77.90.31.11:4556 , piece_id=4GEBIFAQJYJX4KTPHTYUODQZ6ABTTPOA5SYOJBGUV7YDFIHHVO2Q, seg_pos={0 0}, upload=272678]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: expected piece hash; context canceled; EOF Dec 18 14:26:48.541055401 (310h57m48.412138436s) [noise=true , address=77.90.31.11:4556 , piece_id=4GEBIFAQJYJX4KTPHTYUODQZ6ABTTPOA5SYOJBGUV7YDFIHHVO2Q, seg_pos={0 0}, upload=272678]: Long tail cancellation due to optimal threshold Dec 18 14:26:48.541086550 (310h57m48.412169595s) [seg_pos={0 0}, upload=272678]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 18 14:26:48.541155190 (310h57m48.412238226s) [upload=272678]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 18 14:26:48.541159900 (310h57m48.412242935s) [upload=272678]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 18 14:26:48.836195919 (310h57m48.707278954s) [upload=272678]: Done uploading segments... Dec 18 14:26:48.836215697 (310h57m48.707298742s) [upload=272678]: Upload finished. err: <nil> Duration: 5.97284376s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 18 out of 60 is failed (threshold=6) [....X..X.X..............X.......X.....X.X.XX...XXXXX.XX..XX.] duration: 3.722428075s trace: https://jaeger.dev.storj.io/trace/7cbba1e79cb580ce FAIL (2025-12-18 14:31:02 UTC) orbiter warning: context deadline exceeded Duration: 3.722448363s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.................................X....................X.....] duration: 6.278419651s trace: https://jaeger.dev.storj.io/trace/2cbc07c217c30b3f FAIL (2025-12-18 14:30:49 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, Piece ID: O7IY6V6T4UILG3SXQ6K3SPXBVA2VLB22PWVQYIYORGLLYN6LE64A) context deadline exceeded; piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: I2SLWBEKFHZXWYTBZF73DICAZ4ZBXQYU6DES2QWLJMUCFZOCC7CA) context deadline exceeded ======LOGS====== ec 18 14:26:45.953557484 (310h57m45.824640529s) [upload=272678]: Starting upload Dec 18 14:26:45.953631205 (310h57m45.82471425s) [upload=272678]: Done starting upload Dec 18 14:26:45.953727147 (310h57m45.824810202s) [upload=272678]: Uploading segments... Dec 18 14:26:45.953920536 (310h57m45.825003581s) [upload=272678]: Got next segment. Inline: false Dec 18 14:26:45.953955743 (310h57m45.825038798s) [upload=272678]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 18 14:26:45.953963277 (310h57m45.825046322s) [upload=272678]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 18 14:26:46.201381593 (310h57m46.072464628s) [seg_pos={0 0}, upload=272678]: Begin upload segment... Dec 18 14:26:46.201430356 (310h57m46.072513391s) [seg_pos={0 0}, upload=272678]: Done begin upload segment. Dec 18 14:26:46.201523023 (310h57m46.072606088s) [noise=true , address=145.79.227.45:4231 , piece_id=V7B6RPI74SKRJQZCUWFHB2ECD46T3V224VKBL47NIIK5YH6PJCUQ, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.201534584 (310h57m46.072617619s) [noise=true , address=45.91.116.8:4048 , piece_id=BHQYLAJFKTBQ2WGY4XGRI4UOPTGB3SFEUVYZ7XWAC2BKA3ISU3WA, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.203237186 (310h57m46.074320231s) [noise=true , address=45.91.116.14:4190 , piece_id=I2SLWBEKFHZXWYTBZF73DICAZ4ZBXQYU6DES2QWLJMUCFZOCC7CA, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.203669961 (310h57m46.074752996s) [noise=true , address=145.79.227.36:4519 , piece_id=3E54JIZJCR55H36BEOLTDKWRMYA3WHNDPBSALSMNRY45O3BYO6JA, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.203970172 (310h57m46.075053207s) [noise=true , address=145.79.227.38:4978 , piece_id=FRWJ6Z3EH45YRMQVU235FJB75JWK7HOCLZXS7ASITOHLTL3YY5PQ, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.204250205 (310h57m46.07533324s) [noise=true , address=45.91.116.10:4084 , piece_id=UOTQVSUYETMA7UOEURUAT5RPX4QUO4FOXSLQY7UDUKJBCYCOH5BA, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.204534577 (310h57m46.075617612s) [noise=true , address=45.91.116.5:4299 , piece_id=SIZBVTWHVPXBL7OMRRQ467KFLRB56JNA5WDRNSXSNHPSV7UYG5LQ, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.204821995 (310h57m46.07590504s) [noise=true , address=145.79.227.29:4943 , piece_id=4DWWCYWOAU7NSESXDCONC6THXO3TTUYJMN7G6PTG2CP2OOOY2USA, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.206113864 (310h57m46.0771969s) [noise=true , address=77.90.31.11:4556 , piece_id=4GEBIFAQJYJX4KTPHTYUODQZ6ABTTPOA5SYOJBGUV7YDFIHHVO2Q, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.206151877 (310h57m46.077234912s) [noise=true , address=145.79.227.3:4150 , piece_id=XR5E35XW6KBFTREM2GFHARKAAIAINOWMLVYDKKBPFQBXJZOFCC2Q, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.207752204 (310h57m46.078835239s) [noise=true , address=145.79.227.39:4013 , piece_id=2SYWMQU5R7MCTH2E4X732AOVIRVPGUNNXLIN3UFCU7WF4BEJUACQ, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.210366552 (310h57m46.081449597s) [noise=true , address=145.79.227.35:4482 , piece_id=3EGA2CLZBLJWSSULQCBLEEHCMCJDE2FSOY4FCANAQXPEYTWIF53A, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.211613657 (310h57m46.082696692s) [noise=true , address=145.79.227.47:4266 , piece_id=4444BNJTIBUQWLNGTEK3IFD4TIWTCIBCWUHSYBMD3FGZSNDPWT3Q, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.212828561 (310h57m46.083911596s) [noise=true , address=145.79.227.41:4086 , piece_id=DFSNBYWUJ6Z65VTU5ME2IGEDB6KLMTAJM65RC44FUVYHTPEW4ZSA, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.215707031 (310h57m46.086790066s) [noise=true , address=45.91.116.13:4155 , piece_id=O7IY6V6T4UILG3SXQ6K3SPXBVA2VLB22PWVQYIYORGLLYN6LE64A, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.216889753 (310h57m46.087972798s) [noise=true , address=145.79.227.31:4797 , piece_id=F5WHLSRZSSZ4CRJRR2ZPYGGG67SHQ3QMAIHIC42NOVDW2UXARINQ, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:47.912125819 (310h57m47.783208865s) [upload=272678]: Next returned nil segment Dec 18 14:26:47.912143884 (310h57m47.783226929s) [upload=272678]: Waiting for error group managing segments... Dec 18 14:26:48.193212096 (310h57m48.064295131s) [noise=true , address=145.79.227.45:4231 , piece_id=V7B6RPI74SKRJQZCUWFHB2ECD46T3V224VKBL47NIIK5YH6PJCUQ, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.207387668 (310h57m48.078470703s) [noise=true , address=145.79.227.31:4797 , piece_id=F5WHLSRZSSZ4CRJRR2ZPYGGG67SHQ3QMAIHIC42NOVDW2UXARINQ, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.219064714 (310h57m48.090147759s) [noise=true , address=145.79.227.39:4013 , piece_id=2SYWMQU5R7MCTH2E4X732AOVIRVPGUNNXLIN3UFCU7WF4BEJUACQ, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.227887805 (310h57m48.09897085s) [noise=true , address=145.79.227.36:4519 , piece_id=3E54JIZJCR55H36BEOLTDKWRMYA3WHNDPBSALSMNRY45O3BYO6JA, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.229044407 (310h57m48.100127442s) [noise=true , address=145.79.227.41:4086 , piece_id=DFSNBYWUJ6Z65VTU5ME2IGEDB6KLMTAJM65RC44FUVYHTPEW4ZSA, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.234560812 (310h57m48.105643857s) [noise=true , address=145.79.227.47:4266 , piece_id=4444BNJTIBUQWLNGTEK3IFD4TIWTCIBCWUHSYBMD3FGZSNDPWT3Q, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.237048037 (310h57m48.108131072s) [noise=true , address=145.79.227.35:4482 , piece_id=3EGA2CLZBLJWSSULQCBLEEHCMCJDE2FSOY4FCANAQXPEYTWIF53A, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.267241984 (310h57m48.138325019s) [noise=true , address=145.79.227.29:4943 , piece_id=4DWWCYWOAU7NSESXDCONC6THXO3TTUYJMN7G6PTG2CP2OOOY2USA, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.288475576 (310h57m48.159558621s) [noise=true , address=145.79.227.3:4150 , piece_id=XR5E35XW6KBFTREM2GFHARKAAIAINOWMLVYDKKBPFQBXJZOFCC2Q, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.293478393 (310h57m48.164561458s) [noise=true , address=145.79.227.38:4978 , piece_id=FRWJ6Z3EH45YRMQVU235FJB75JWK7HOCLZXS7ASITOHLTL3YY5PQ, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.423759785 (310h57m48.29484282s) [noise=true , address=45.91.116.14:4190 , piece_id=I2SLWBEKFHZXWYTBZF73DICAZ4ZBXQYU6DES2QWLJMUCFZOCC7CA, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.485697347 (310h57m48.356780392s) [noise=true , address=45.91.116.10:4084 , piece_id=UOTQVSUYETMA7UOEURUAT5RPX4QUO4FOXSLQY7UDUKJBCYCOH5BA, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.540540711 (310h57m48.411623756s) [noise=true , address=45.91.116.13:4155 , piece_id=O7IY6V6T4UILG3SXQ6K3SPXBVA2VLB22PWVQYIYORGLLYN6LE64A, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.540575787 (310h57m48.411658832s) [seg_pos={0 0}, upload=272678]: Segment reached optimal threshold of 13 pieces. Dec 18 14:26:48.540831424 (310h57m48.411914459s) [noise=true , address=45.91.116.8:4048 , piece_id=BHQYLAJFKTBQ2WGY4XGRI4UOPTGB3SFEUVYZ7XWAC2BKA3ISU3WA, seg_pos={0 0}, upload=272678]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: write tcp 188.166.225.170:49328->45.91.116.8:4048: use of closed network connection; write tcp 188.166.225.170:49328->45.91.116.8:4048: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49328->45.91.116.8:4048: use of closed network connection Dec 18 14:26:48.540840511 (310h57m48.411923556s) [noise=true , address=45.91.116.8:4048 , piece_id=BHQYLAJFKTBQ2WGY4XGRI4UOPTGB3SFEUVYZ7XWAC2BKA3ISU3WA, seg_pos={0 0}, upload=272678]: Long tail cancellation due to optimal threshold Dec 18 14:26:48.540888052 (310h57m48.411971107s) [noise=true , address=45.91.116.5:4299 , piece_id=SIZBVTWHVPXBL7OMRRQ467KFLRB56JNA5WDRNSXSNHPSV7UYG5LQ, seg_pos={0 0}, upload=272678]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Dec 18 14:26:48.540899453 (310h57m48.411982498s) [noise=true , address=45.91.116.5:4299 , piece_id=SIZBVTWHVPXBL7OMRRQ467KFLRB56JNA5WDRNSXSNHPSV7UYG5LQ, seg_pos={0 0}, upload=272678]: Long tail cancellation due to optimal threshold Dec 18 14:26:48.541046764 (310h57m48.412129809s) [noise=true , address=77.90.31.11:4556 , piece_id=4GEBIFAQJYJX4KTPHTYUODQZ6ABTTPOA5SYOJBGUV7YDFIHHVO2Q, seg_pos={0 0}, upload=272678]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: expected piece hash; context canceled; EOF Dec 18 14:26:48.541055401 (310h57m48.412138436s) [noise=true , address=77.90.31.11:4556 , piece_id=4GEBIFAQJYJX4KTPHTYUODQZ6ABTTPOA5SYOJBGUV7YDFIHHVO2Q, seg_pos={0 0}, upload=272678]: Long tail cancellation due to optimal threshold Dec 18 14:26:48.541086550 (310h57m48.412169595s) [seg_pos={0 0}, upload=272678]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 18 14:26:48.541155190 (310h57m48.412238226s) [upload=272678]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 18 14:26:48.541159900 (310h57m48.412242935s) [upload=272678]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 18 14:26:48.836195919 (310h57m48.707278954s) [upload=272678]: Done uploading segments... Dec 18 14:26:48.836215697 (310h57m48.707298742s) [upload=272678]: Upload finished. err: <nil> Duration: 6.278450889s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..................................X....................X....] duration: 5.777347354s trace: https://jaeger.dev.storj.io/trace/2cbc07c217c30b3f FAIL (2025-12-18 14:30:20 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, Piece ID: O7IY6V6T4UILG3SXQ6K3SPXBVA2VLB22PWVQYIYORGLLYN6LE64A) context deadline exceeded; piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: I2SLWBEKFHZXWYTBZF73DICAZ4ZBXQYU6DES2QWLJMUCFZOCC7CA) context deadline exceeded ======LOGS====== ec 18 14:26:45.953557484 (310h57m45.824640529s) [upload=272678]: Starting upload Dec 18 14:26:45.953631205 (310h57m45.82471425s) [upload=272678]: Done starting upload Dec 18 14:26:45.953727147 (310h57m45.824810202s) [upload=272678]: Uploading segments... Dec 18 14:26:45.953920536 (310h57m45.825003581s) [upload=272678]: Got next segment. Inline: false Dec 18 14:26:45.953955743 (310h57m45.825038798s) [upload=272678]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 18 14:26:45.953963277 (310h57m45.825046322s) [upload=272678]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 18 14:26:46.201381593 (310h57m46.072464628s) [seg_pos={0 0}, upload=272678]: Begin upload segment... Dec 18 14:26:46.201430356 (310h57m46.072513391s) [seg_pos={0 0}, upload=272678]: Done begin upload segment. Dec 18 14:26:46.201523023 (310h57m46.072606088s) [noise=true , address=145.79.227.45:4231 , piece_id=V7B6RPI74SKRJQZCUWFHB2ECD46T3V224VKBL47NIIK5YH6PJCUQ, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.201534584 (310h57m46.072617619s) [noise=true , address=45.91.116.8:4048 , piece_id=BHQYLAJFKTBQ2WGY4XGRI4UOPTGB3SFEUVYZ7XWAC2BKA3ISU3WA, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.203237186 (310h57m46.074320231s) [noise=true , address=45.91.116.14:4190 , piece_id=I2SLWBEKFHZXWYTBZF73DICAZ4ZBXQYU6DES2QWLJMUCFZOCC7CA, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.203669961 (310h57m46.074752996s) [noise=true , address=145.79.227.36:4519 , piece_id=3E54JIZJCR55H36BEOLTDKWRMYA3WHNDPBSALSMNRY45O3BYO6JA, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.203970172 (310h57m46.075053207s) [noise=true , address=145.79.227.38:4978 , piece_id=FRWJ6Z3EH45YRMQVU235FJB75JWK7HOCLZXS7ASITOHLTL3YY5PQ, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.204250205 (310h57m46.07533324s) [noise=true , address=45.91.116.10:4084 , piece_id=UOTQVSUYETMA7UOEURUAT5RPX4QUO4FOXSLQY7UDUKJBCYCOH5BA, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.204534577 (310h57m46.075617612s) [noise=true , address=45.91.116.5:4299 , piece_id=SIZBVTWHVPXBL7OMRRQ467KFLRB56JNA5WDRNSXSNHPSV7UYG5LQ, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.204821995 (310h57m46.07590504s) [noise=true , address=145.79.227.29:4943 , piece_id=4DWWCYWOAU7NSESXDCONC6THXO3TTUYJMN7G6PTG2CP2OOOY2USA, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.206113864 (310h57m46.0771969s) [noise=true , address=77.90.31.11:4556 , piece_id=4GEBIFAQJYJX4KTPHTYUODQZ6ABTTPOA5SYOJBGUV7YDFIHHVO2Q, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.206151877 (310h57m46.077234912s) [noise=true , address=145.79.227.3:4150 , piece_id=XR5E35XW6KBFTREM2GFHARKAAIAINOWMLVYDKKBPFQBXJZOFCC2Q, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.207752204 (310h57m46.078835239s) [noise=true , address=145.79.227.39:4013 , piece_id=2SYWMQU5R7MCTH2E4X732AOVIRVPGUNNXLIN3UFCU7WF4BEJUACQ, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.210366552 (310h57m46.081449597s) [noise=true , address=145.79.227.35:4482 , piece_id=3EGA2CLZBLJWSSULQCBLEEHCMCJDE2FSOY4FCANAQXPEYTWIF53A, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.211613657 (310h57m46.082696692s) [noise=true , address=145.79.227.47:4266 , piece_id=4444BNJTIBUQWLNGTEK3IFD4TIWTCIBCWUHSYBMD3FGZSNDPWT3Q, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.212828561 (310h57m46.083911596s) [noise=true , address=145.79.227.41:4086 , piece_id=DFSNBYWUJ6Z65VTU5ME2IGEDB6KLMTAJM65RC44FUVYHTPEW4ZSA, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.215707031 (310h57m46.086790066s) [noise=true , address=45.91.116.13:4155 , piece_id=O7IY6V6T4UILG3SXQ6K3SPXBVA2VLB22PWVQYIYORGLLYN6LE64A, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.216889753 (310h57m46.087972798s) [noise=true , address=145.79.227.31:4797 , piece_id=F5WHLSRZSSZ4CRJRR2ZPYGGG67SHQ3QMAIHIC42NOVDW2UXARINQ, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:47.912125819 (310h57m47.783208865s) [upload=272678]: Next returned nil segment Dec 18 14:26:47.912143884 (310h57m47.783226929s) [upload=272678]: Waiting for error group managing segments... Dec 18 14:26:48.193212096 (310h57m48.064295131s) [noise=true , address=145.79.227.45:4231 , piece_id=V7B6RPI74SKRJQZCUWFHB2ECD46T3V224VKBL47NIIK5YH6PJCUQ, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.207387668 (310h57m48.078470703s) [noise=true , address=145.79.227.31:4797 , piece_id=F5WHLSRZSSZ4CRJRR2ZPYGGG67SHQ3QMAIHIC42NOVDW2UXARINQ, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.219064714 (310h57m48.090147759s) [noise=true , address=145.79.227.39:4013 , piece_id=2SYWMQU5R7MCTH2E4X732AOVIRVPGUNNXLIN3UFCU7WF4BEJUACQ, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.227887805 (310h57m48.09897085s) [noise=true , address=145.79.227.36:4519 , piece_id=3E54JIZJCR55H36BEOLTDKWRMYA3WHNDPBSALSMNRY45O3BYO6JA, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.229044407 (310h57m48.100127442s) [noise=true , address=145.79.227.41:4086 , piece_id=DFSNBYWUJ6Z65VTU5ME2IGEDB6KLMTAJM65RC44FUVYHTPEW4ZSA, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.234560812 (310h57m48.105643857s) [noise=true , address=145.79.227.47:4266 , piece_id=4444BNJTIBUQWLNGTEK3IFD4TIWTCIBCWUHSYBMD3FGZSNDPWT3Q, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.237048037 (310h57m48.108131072s) [noise=true , address=145.79.227.35:4482 , piece_id=3EGA2CLZBLJWSSULQCBLEEHCMCJDE2FSOY4FCANAQXPEYTWIF53A, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.267241984 (310h57m48.138325019s) [noise=true , address=145.79.227.29:4943 , piece_id=4DWWCYWOAU7NSESXDCONC6THXO3TTUYJMN7G6PTG2CP2OOOY2USA, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.288475576 (310h57m48.159558621s) [noise=true , address=145.79.227.3:4150 , piece_id=XR5E35XW6KBFTREM2GFHARKAAIAINOWMLVYDKKBPFQBXJZOFCC2Q, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.293478393 (310h57m48.164561458s) [noise=true , address=145.79.227.38:4978 , piece_id=FRWJ6Z3EH45YRMQVU235FJB75JWK7HOCLZXS7ASITOHLTL3YY5PQ, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.423759785 (310h57m48.29484282s) [noise=true , address=45.91.116.14:4190 , piece_id=I2SLWBEKFHZXWYTBZF73DICAZ4ZBXQYU6DES2QWLJMUCFZOCC7CA, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.485697347 (310h57m48.356780392s) [noise=true , address=45.91.116.10:4084 , piece_id=UOTQVSUYETMA7UOEURUAT5RPX4QUO4FOXSLQY7UDUKJBCYCOH5BA, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.540540711 (310h57m48.411623756s) [noise=true , address=45.91.116.13:4155 , piece_id=O7IY6V6T4UILG3SXQ6K3SPXBVA2VLB22PWVQYIYORGLLYN6LE64A, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.540575787 (310h57m48.411658832s) [seg_pos={0 0}, upload=272678]: Segment reached optimal threshold of 13 pieces. Dec 18 14:26:48.540831424 (310h57m48.411914459s) [noise=true , address=45.91.116.8:4048 , piece_id=BHQYLAJFKTBQ2WGY4XGRI4UOPTGB3SFEUVYZ7XWAC2BKA3ISU3WA, seg_pos={0 0}, upload=272678]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: write tcp 188.166.225.170:49328->45.91.116.8:4048: use of closed network connection; write tcp 188.166.225.170:49328->45.91.116.8:4048: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49328->45.91.116.8:4048: use of closed network connection Dec 18 14:26:48.540840511 (310h57m48.411923556s) [noise=true , address=45.91.116.8:4048 , piece_id=BHQYLAJFKTBQ2WGY4XGRI4UOPTGB3SFEUVYZ7XWAC2BKA3ISU3WA, seg_pos={0 0}, upload=272678]: Long tail cancellation due to optimal threshold Dec 18 14:26:48.540888052 (310h57m48.411971107s) [noise=true , address=45.91.116.5:4299 , piece_id=SIZBVTWHVPXBL7OMRRQ467KFLRB56JNA5WDRNSXSNHPSV7UYG5LQ, seg_pos={0 0}, upload=272678]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Dec 18 14:26:48.540899453 (310h57m48.411982498s) [noise=true , address=45.91.116.5:4299 , piece_id=SIZBVTWHVPXBL7OMRRQ467KFLRB56JNA5WDRNSXSNHPSV7UYG5LQ, seg_pos={0 0}, upload=272678]: Long tail cancellation due to optimal threshold Dec 18 14:26:48.541046764 (310h57m48.412129809s) [noise=true , address=77.90.31.11:4556 , piece_id=4GEBIFAQJYJX4KTPHTYUODQZ6ABTTPOA5SYOJBGUV7YDFIHHVO2Q, seg_pos={0 0}, upload=272678]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: expected piece hash; context canceled; EOF Dec 18 14:26:48.541055401 (310h57m48.412138436s) [noise=true , address=77.90.31.11:4556 , piece_id=4GEBIFAQJYJX4KTPHTYUODQZ6ABTTPOA5SYOJBGUV7YDFIHHVO2Q, seg_pos={0 0}, upload=272678]: Long tail cancellation due to optimal threshold Dec 18 14:26:48.541086550 (310h57m48.412169595s) [seg_pos={0 0}, upload=272678]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 18 14:26:48.541155190 (310h57m48.412238226s) [upload=272678]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 18 14:26:48.541159900 (310h57m48.412242935s) [upload=272678]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 18 14:26:48.836195919 (310h57m48.707278954s) [upload=272678]: Done uploading segments... Dec 18 14:26:48.836215697 (310h57m48.707298742s) [upload=272678]: Upload finished. err: <nil> Duration: 5.777379685s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 18 out of 60 is failed (threshold=6) [.....X..X.X..............X.......X.....X.X.XX...XXXXX.XX..XX] duration: 5.001607072s trace: https://jaeger.dev.storj.io/trace/7cbba1e79cb580ce FAIL (2025-12-18 14:30:02 UTC) orbiter warning: context deadline exceeded Duration: 5.001625417s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...................................X....................X...] duration: 5.83486233s trace: https://jaeger.dev.storj.io/trace/2cbc07c217c30b3f FAIL (2025-12-18 14:29:30 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, Piece ID: O7IY6V6T4UILG3SXQ6K3SPXBVA2VLB22PWVQYIYORGLLYN6LE64A) context deadline exceeded; piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: I2SLWBEKFHZXWYTBZF73DICAZ4ZBXQYU6DES2QWLJMUCFZOCC7CA) context deadline exceeded ======LOGS====== ec 18 14:26:45.953557484 (310h57m45.824640529s) [upload=272678]: Starting upload Dec 18 14:26:45.953631205 (310h57m45.82471425s) [upload=272678]: Done starting upload Dec 18 14:26:45.953727147 (310h57m45.824810202s) [upload=272678]: Uploading segments... Dec 18 14:26:45.953920536 (310h57m45.825003581s) [upload=272678]: Got next segment. Inline: false Dec 18 14:26:45.953955743 (310h57m45.825038798s) [upload=272678]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 18 14:26:45.953963277 (310h57m45.825046322s) [upload=272678]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 18 14:26:46.201381593 (310h57m46.072464628s) [seg_pos={0 0}, upload=272678]: Begin upload segment... Dec 18 14:26:46.201430356 (310h57m46.072513391s) [seg_pos={0 0}, upload=272678]: Done begin upload segment. Dec 18 14:26:46.201523023 (310h57m46.072606088s) [noise=true , address=145.79.227.45:4231 , piece_id=V7B6RPI74SKRJQZCUWFHB2ECD46T3V224VKBL47NIIK5YH6PJCUQ, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.201534584 (310h57m46.072617619s) [noise=true , address=45.91.116.8:4048 , piece_id=BHQYLAJFKTBQ2WGY4XGRI4UOPTGB3SFEUVYZ7XWAC2BKA3ISU3WA, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.203237186 (310h57m46.074320231s) [noise=true , address=45.91.116.14:4190 , piece_id=I2SLWBEKFHZXWYTBZF73DICAZ4ZBXQYU6DES2QWLJMUCFZOCC7CA, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.203669961 (310h57m46.074752996s) [noise=true , address=145.79.227.36:4519 , piece_id=3E54JIZJCR55H36BEOLTDKWRMYA3WHNDPBSALSMNRY45O3BYO6JA, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.203970172 (310h57m46.075053207s) [noise=true , address=145.79.227.38:4978 , piece_id=FRWJ6Z3EH45YRMQVU235FJB75JWK7HOCLZXS7ASITOHLTL3YY5PQ, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.204250205 (310h57m46.07533324s) [noise=true , address=45.91.116.10:4084 , piece_id=UOTQVSUYETMA7UOEURUAT5RPX4QUO4FOXSLQY7UDUKJBCYCOH5BA, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.204534577 (310h57m46.075617612s) [noise=true , address=45.91.116.5:4299 , piece_id=SIZBVTWHVPXBL7OMRRQ467KFLRB56JNA5WDRNSXSNHPSV7UYG5LQ, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.204821995 (310h57m46.07590504s) [noise=true , address=145.79.227.29:4943 , piece_id=4DWWCYWOAU7NSESXDCONC6THXO3TTUYJMN7G6PTG2CP2OOOY2USA, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.206113864 (310h57m46.0771969s) [noise=true , address=77.90.31.11:4556 , piece_id=4GEBIFAQJYJX4KTPHTYUODQZ6ABTTPOA5SYOJBGUV7YDFIHHVO2Q, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.206151877 (310h57m46.077234912s) [noise=true , address=145.79.227.3:4150 , piece_id=XR5E35XW6KBFTREM2GFHARKAAIAINOWMLVYDKKBPFQBXJZOFCC2Q, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.207752204 (310h57m46.078835239s) [noise=true , address=145.79.227.39:4013 , piece_id=2SYWMQU5R7MCTH2E4X732AOVIRVPGUNNXLIN3UFCU7WF4BEJUACQ, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.210366552 (310h57m46.081449597s) [noise=true , address=145.79.227.35:4482 , piece_id=3EGA2CLZBLJWSSULQCBLEEHCMCJDE2FSOY4FCANAQXPEYTWIF53A, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.211613657 (310h57m46.082696692s) [noise=true , address=145.79.227.47:4266 , piece_id=4444BNJTIBUQWLNGTEK3IFD4TIWTCIBCWUHSYBMD3FGZSNDPWT3Q, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.212828561 (310h57m46.083911596s) [noise=true , address=145.79.227.41:4086 , piece_id=DFSNBYWUJ6Z65VTU5ME2IGEDB6KLMTAJM65RC44FUVYHTPEW4ZSA, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.215707031 (310h57m46.086790066s) [noise=true , address=45.91.116.13:4155 , piece_id=O7IY6V6T4UILG3SXQ6K3SPXBVA2VLB22PWVQYIYORGLLYN6LE64A, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.216889753 (310h57m46.087972798s) [noise=true , address=145.79.227.31:4797 , piece_id=F5WHLSRZSSZ4CRJRR2ZPYGGG67SHQ3QMAIHIC42NOVDW2UXARINQ, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:47.912125819 (310h57m47.783208865s) [upload=272678]: Next returned nil segment Dec 18 14:26:47.912143884 (310h57m47.783226929s) [upload=272678]: Waiting for error group managing segments... Dec 18 14:26:48.193212096 (310h57m48.064295131s) [noise=true , address=145.79.227.45:4231 , piece_id=V7B6RPI74SKRJQZCUWFHB2ECD46T3V224VKBL47NIIK5YH6PJCUQ, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.207387668 (310h57m48.078470703s) [noise=true , address=145.79.227.31:4797 , piece_id=F5WHLSRZSSZ4CRJRR2ZPYGGG67SHQ3QMAIHIC42NOVDW2UXARINQ, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.219064714 (310h57m48.090147759s) [noise=true , address=145.79.227.39:4013 , piece_id=2SYWMQU5R7MCTH2E4X732AOVIRVPGUNNXLIN3UFCU7WF4BEJUACQ, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.227887805 (310h57m48.09897085s) [noise=true , address=145.79.227.36:4519 , piece_id=3E54JIZJCR55H36BEOLTDKWRMYA3WHNDPBSALSMNRY45O3BYO6JA, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.229044407 (310h57m48.100127442s) [noise=true , address=145.79.227.41:4086 , piece_id=DFSNBYWUJ6Z65VTU5ME2IGEDB6KLMTAJM65RC44FUVYHTPEW4ZSA, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.234560812 (310h57m48.105643857s) [noise=true , address=145.79.227.47:4266 , piece_id=4444BNJTIBUQWLNGTEK3IFD4TIWTCIBCWUHSYBMD3FGZSNDPWT3Q, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.237048037 (310h57m48.108131072s) [noise=true , address=145.79.227.35:4482 , piece_id=3EGA2CLZBLJWSSULQCBLEEHCMCJDE2FSOY4FCANAQXPEYTWIF53A, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.267241984 (310h57m48.138325019s) [noise=true , address=145.79.227.29:4943 , piece_id=4DWWCYWOAU7NSESXDCONC6THXO3TTUYJMN7G6PTG2CP2OOOY2USA, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.288475576 (310h57m48.159558621s) [noise=true , address=145.79.227.3:4150 , piece_id=XR5E35XW6KBFTREM2GFHARKAAIAINOWMLVYDKKBPFQBXJZOFCC2Q, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.293478393 (310h57m48.164561458s) [noise=true , address=145.79.227.38:4978 , piece_id=FRWJ6Z3EH45YRMQVU235FJB75JWK7HOCLZXS7ASITOHLTL3YY5PQ, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.423759785 (310h57m48.29484282s) [noise=true , address=45.91.116.14:4190 , piece_id=I2SLWBEKFHZXWYTBZF73DICAZ4ZBXQYU6DES2QWLJMUCFZOCC7CA, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.485697347 (310h57m48.356780392s) [noise=true , address=45.91.116.10:4084 , piece_id=UOTQVSUYETMA7UOEURUAT5RPX4QUO4FOXSLQY7UDUKJBCYCOH5BA, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.540540711 (310h57m48.411623756s) [noise=true , address=45.91.116.13:4155 , piece_id=O7IY6V6T4UILG3SXQ6K3SPXBVA2VLB22PWVQYIYORGLLYN6LE64A, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.540575787 (310h57m48.411658832s) [seg_pos={0 0}, upload=272678]: Segment reached optimal threshold of 13 pieces. Dec 18 14:26:48.540831424 (310h57m48.411914459s) [noise=true , address=45.91.116.8:4048 , piece_id=BHQYLAJFKTBQ2WGY4XGRI4UOPTGB3SFEUVYZ7XWAC2BKA3ISU3WA, seg_pos={0 0}, upload=272678]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: write tcp 188.166.225.170:49328->45.91.116.8:4048: use of closed network connection; write tcp 188.166.225.170:49328->45.91.116.8:4048: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49328->45.91.116.8:4048: use of closed network connection Dec 18 14:26:48.540840511 (310h57m48.411923556s) [noise=true , address=45.91.116.8:4048 , piece_id=BHQYLAJFKTBQ2WGY4XGRI4UOPTGB3SFEUVYZ7XWAC2BKA3ISU3WA, seg_pos={0 0}, upload=272678]: Long tail cancellation due to optimal threshold Dec 18 14:26:48.540888052 (310h57m48.411971107s) [noise=true , address=45.91.116.5:4299 , piece_id=SIZBVTWHVPXBL7OMRRQ467KFLRB56JNA5WDRNSXSNHPSV7UYG5LQ, seg_pos={0 0}, upload=272678]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Dec 18 14:26:48.540899453 (310h57m48.411982498s) [noise=true , address=45.91.116.5:4299 , piece_id=SIZBVTWHVPXBL7OMRRQ467KFLRB56JNA5WDRNSXSNHPSV7UYG5LQ, seg_pos={0 0}, upload=272678]: Long tail cancellation due to optimal threshold Dec 18 14:26:48.541046764 (310h57m48.412129809s) [noise=true , address=77.90.31.11:4556 , piece_id=4GEBIFAQJYJX4KTPHTYUODQZ6ABTTPOA5SYOJBGUV7YDFIHHVO2Q, seg_pos={0 0}, upload=272678]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: expected piece hash; context canceled; EOF Dec 18 14:26:48.541055401 (310h57m48.412138436s) [noise=true , address=77.90.31.11:4556 , piece_id=4GEBIFAQJYJX4KTPHTYUODQZ6ABTTPOA5SYOJBGUV7YDFIHHVO2Q, seg_pos={0 0}, upload=272678]: Long tail cancellation due to optimal threshold Dec 18 14:26:48.541086550 (310h57m48.412169595s) [seg_pos={0 0}, upload=272678]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 18 14:26:48.541155190 (310h57m48.412238226s) [upload=272678]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 18 14:26:48.541159900 (310h57m48.412242935s) [upload=272678]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 18 14:26:48.836195919 (310h57m48.707278954s) [upload=272678]: Done uploading segments... Dec 18 14:26:48.836215697 (310h57m48.707298742s) [upload=272678]: Upload finished. err: <nil> Duration: 5.834881849s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 18 out of 60 is failed (threshold=6) [X.....X..X.X..............X.......X.....X.X.XX...XXXXX.XX..X] duration: 5.001675014s trace: https://jaeger.dev.storj.io/trace/293ea3611507c5d FAIL (2025-12-18 14:29:15 UTC) orbiter warning: context deadline exceeded Duration: 5.001696916s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 18 out of 60 is failed (threshold=6) [XX.....X..X.X..............X.......X.....X.X.XX...XXXXX.XX..] duration: 4.532900207s trace: https://jaeger.dev.storj.io/trace/52122bfeccd0cf7e FAIL (2025-12-18 14:28:48 UTC) orbiter warning: context deadline exceeded Duration: 4.532927531s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [....................................X....................X..] duration: 5.791670639s trace: https://jaeger.dev.storj.io/trace/2cbc07c217c30b3f FAIL (2025-12-18 14:28:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, Piece ID: O7IY6V6T4UILG3SXQ6K3SPXBVA2VLB22PWVQYIYORGLLYN6LE64A) context deadline exceeded; piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: I2SLWBEKFHZXWYTBZF73DICAZ4ZBXQYU6DES2QWLJMUCFZOCC7CA) context deadline exceeded ======LOGS====== ec 18 14:26:45.953557484 (310h57m45.824640529s) [upload=272678]: Starting upload Dec 18 14:26:45.953631205 (310h57m45.82471425s) [upload=272678]: Done starting upload Dec 18 14:26:45.953727147 (310h57m45.824810202s) [upload=272678]: Uploading segments... Dec 18 14:26:45.953920536 (310h57m45.825003581s) [upload=272678]: Got next segment. Inline: false Dec 18 14:26:45.953955743 (310h57m45.825038798s) [upload=272678]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 18 14:26:45.953963277 (310h57m45.825046322s) [upload=272678]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 18 14:26:46.201381593 (310h57m46.072464628s) [seg_pos={0 0}, upload=272678]: Begin upload segment... Dec 18 14:26:46.201430356 (310h57m46.072513391s) [seg_pos={0 0}, upload=272678]: Done begin upload segment. Dec 18 14:26:46.201523023 (310h57m46.072606088s) [noise=true , address=145.79.227.45:4231 , piece_id=V7B6RPI74SKRJQZCUWFHB2ECD46T3V224VKBL47NIIK5YH6PJCUQ, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.201534584 (310h57m46.072617619s) [noise=true , address=45.91.116.8:4048 , piece_id=BHQYLAJFKTBQ2WGY4XGRI4UOPTGB3SFEUVYZ7XWAC2BKA3ISU3WA, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.203237186 (310h57m46.074320231s) [noise=true , address=45.91.116.14:4190 , piece_id=I2SLWBEKFHZXWYTBZF73DICAZ4ZBXQYU6DES2QWLJMUCFZOCC7CA, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.203669961 (310h57m46.074752996s) [noise=true , address=145.79.227.36:4519 , piece_id=3E54JIZJCR55H36BEOLTDKWRMYA3WHNDPBSALSMNRY45O3BYO6JA, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.203970172 (310h57m46.075053207s) [noise=true , address=145.79.227.38:4978 , piece_id=FRWJ6Z3EH45YRMQVU235FJB75JWK7HOCLZXS7ASITOHLTL3YY5PQ, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.204250205 (310h57m46.07533324s) [noise=true , address=45.91.116.10:4084 , piece_id=UOTQVSUYETMA7UOEURUAT5RPX4QUO4FOXSLQY7UDUKJBCYCOH5BA, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.204534577 (310h57m46.075617612s) [noise=true , address=45.91.116.5:4299 , piece_id=SIZBVTWHVPXBL7OMRRQ467KFLRB56JNA5WDRNSXSNHPSV7UYG5LQ, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.204821995 (310h57m46.07590504s) [noise=true , address=145.79.227.29:4943 , piece_id=4DWWCYWOAU7NSESXDCONC6THXO3TTUYJMN7G6PTG2CP2OOOY2USA, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.206113864 (310h57m46.0771969s) [noise=true , address=77.90.31.11:4556 , piece_id=4GEBIFAQJYJX4KTPHTYUODQZ6ABTTPOA5SYOJBGUV7YDFIHHVO2Q, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.206151877 (310h57m46.077234912s) [noise=true , address=145.79.227.3:4150 , piece_id=XR5E35XW6KBFTREM2GFHARKAAIAINOWMLVYDKKBPFQBXJZOFCC2Q, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.207752204 (310h57m46.078835239s) [noise=true , address=145.79.227.39:4013 , piece_id=2SYWMQU5R7MCTH2E4X732AOVIRVPGUNNXLIN3UFCU7WF4BEJUACQ, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.210366552 (310h57m46.081449597s) [noise=true , address=145.79.227.35:4482 , piece_id=3EGA2CLZBLJWSSULQCBLEEHCMCJDE2FSOY4FCANAQXPEYTWIF53A, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.211613657 (310h57m46.082696692s) [noise=true , address=145.79.227.47:4266 , piece_id=4444BNJTIBUQWLNGTEK3IFD4TIWTCIBCWUHSYBMD3FGZSNDPWT3Q, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.212828561 (310h57m46.083911596s) [noise=true , address=145.79.227.41:4086 , piece_id=DFSNBYWUJ6Z65VTU5ME2IGEDB6KLMTAJM65RC44FUVYHTPEW4ZSA, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.215707031 (310h57m46.086790066s) [noise=true , address=45.91.116.13:4155 , piece_id=O7IY6V6T4UILG3SXQ6K3SPXBVA2VLB22PWVQYIYORGLLYN6LE64A, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.216889753 (310h57m46.087972798s) [noise=true , address=145.79.227.31:4797 , piece_id=F5WHLSRZSSZ4CRJRR2ZPYGGG67SHQ3QMAIHIC42NOVDW2UXARINQ, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:47.912125819 (310h57m47.783208865s) [upload=272678]: Next returned nil segment Dec 18 14:26:47.912143884 (310h57m47.783226929s) [upload=272678]: Waiting for error group managing segments... Dec 18 14:26:48.193212096 (310h57m48.064295131s) [noise=true , address=145.79.227.45:4231 , piece_id=V7B6RPI74SKRJQZCUWFHB2ECD46T3V224VKBL47NIIK5YH6PJCUQ, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.207387668 (310h57m48.078470703s) [noise=true , address=145.79.227.31:4797 , piece_id=F5WHLSRZSSZ4CRJRR2ZPYGGG67SHQ3QMAIHIC42NOVDW2UXARINQ, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.219064714 (310h57m48.090147759s) [noise=true , address=145.79.227.39:4013 , piece_id=2SYWMQU5R7MCTH2E4X732AOVIRVPGUNNXLIN3UFCU7WF4BEJUACQ, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.227887805 (310h57m48.09897085s) [noise=true , address=145.79.227.36:4519 , piece_id=3E54JIZJCR55H36BEOLTDKWRMYA3WHNDPBSALSMNRY45O3BYO6JA, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.229044407 (310h57m48.100127442s) [noise=true , address=145.79.227.41:4086 , piece_id=DFSNBYWUJ6Z65VTU5ME2IGEDB6KLMTAJM65RC44FUVYHTPEW4ZSA, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.234560812 (310h57m48.105643857s) [noise=true , address=145.79.227.47:4266 , piece_id=4444BNJTIBUQWLNGTEK3IFD4TIWTCIBCWUHSYBMD3FGZSNDPWT3Q, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.237048037 (310h57m48.108131072s) [noise=true , address=145.79.227.35:4482 , piece_id=3EGA2CLZBLJWSSULQCBLEEHCMCJDE2FSOY4FCANAQXPEYTWIF53A, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.267241984 (310h57m48.138325019s) [noise=true , address=145.79.227.29:4943 , piece_id=4DWWCYWOAU7NSESXDCONC6THXO3TTUYJMN7G6PTG2CP2OOOY2USA, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.288475576 (310h57m48.159558621s) [noise=true , address=145.79.227.3:4150 , piece_id=XR5E35XW6KBFTREM2GFHARKAAIAINOWMLVYDKKBPFQBXJZOFCC2Q, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.293478393 (310h57m48.164561458s) [noise=true , address=145.79.227.38:4978 , piece_id=FRWJ6Z3EH45YRMQVU235FJB75JWK7HOCLZXS7ASITOHLTL3YY5PQ, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.423759785 (310h57m48.29484282s) [noise=true , address=45.91.116.14:4190 , piece_id=I2SLWBEKFHZXWYTBZF73DICAZ4ZBXQYU6DES2QWLJMUCFZOCC7CA, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.485697347 (310h57m48.356780392s) [noise=true , address=45.91.116.10:4084 , piece_id=UOTQVSUYETMA7UOEURUAT5RPX4QUO4FOXSLQY7UDUKJBCYCOH5BA, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.540540711 (310h57m48.411623756s) [noise=true , address=45.91.116.13:4155 , piece_id=O7IY6V6T4UILG3SXQ6K3SPXBVA2VLB22PWVQYIYORGLLYN6LE64A, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.540575787 (310h57m48.411658832s) [seg_pos={0 0}, upload=272678]: Segment reached optimal threshold of 13 pieces. Dec 18 14:26:48.540831424 (310h57m48.411914459s) [noise=true , address=45.91.116.8:4048 , piece_id=BHQYLAJFKTBQ2WGY4XGRI4UOPTGB3SFEUVYZ7XWAC2BKA3ISU3WA, seg_pos={0 0}, upload=272678]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: write tcp 188.166.225.170:49328->45.91.116.8:4048: use of closed network connection; write tcp 188.166.225.170:49328->45.91.116.8:4048: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49328->45.91.116.8:4048: use of closed network connection Dec 18 14:26:48.540840511 (310h57m48.411923556s) [noise=true , address=45.91.116.8:4048 , piece_id=BHQYLAJFKTBQ2WGY4XGRI4UOPTGB3SFEUVYZ7XWAC2BKA3ISU3WA, seg_pos={0 0}, upload=272678]: Long tail cancellation due to optimal threshold Dec 18 14:26:48.540888052 (310h57m48.411971107s) [noise=true , address=45.91.116.5:4299 , piece_id=SIZBVTWHVPXBL7OMRRQ467KFLRB56JNA5WDRNSXSNHPSV7UYG5LQ, seg_pos={0 0}, upload=272678]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Dec 18 14:26:48.540899453 (310h57m48.411982498s) [noise=true , address=45.91.116.5:4299 , piece_id=SIZBVTWHVPXBL7OMRRQ467KFLRB56JNA5WDRNSXSNHPSV7UYG5LQ, seg_pos={0 0}, upload=272678]: Long tail cancellation due to optimal threshold Dec 18 14:26:48.541046764 (310h57m48.412129809s) [noise=true , address=77.90.31.11:4556 , piece_id=4GEBIFAQJYJX4KTPHTYUODQZ6ABTTPOA5SYOJBGUV7YDFIHHVO2Q, seg_pos={0 0}, upload=272678]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: expected piece hash; context canceled; EOF Dec 18 14:26:48.541055401 (310h57m48.412138436s) [noise=true , address=77.90.31.11:4556 , piece_id=4GEBIFAQJYJX4KTPHTYUODQZ6ABTTPOA5SYOJBGUV7YDFIHHVO2Q, seg_pos={0 0}, upload=272678]: Long tail cancellation due to optimal threshold Dec 18 14:26:48.541086550 (310h57m48.412169595s) [seg_pos={0 0}, upload=272678]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 18 14:26:48.541155190 (310h57m48.412238226s) [upload=272678]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 18 14:26:48.541159900 (310h57m48.412242935s) [upload=272678]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 18 14:26:48.836195919 (310h57m48.707278954s) [upload=272678]: Done uploading segments... Dec 18 14:26:48.836215697 (310h57m48.707298742s) [upload=272678]: Upload finished. err: <nil> Duration: 5.791700406s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 19 out of 60 is failed (threshold=6) [XXX.....X..X.X..............X.......X.....X.X.XX...XXXXX.XX.] duration: 4.752053157s trace: https://jaeger.dev.storj.io/trace/52122bfeccd0cf7e FAIL (2025-12-18 14:28:23 UTC) orbiter warning: context deadline exceeded Duration: 4.752077554s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....................................X....................X.] duration: 5.841057061s trace: https://jaeger.dev.storj.io/trace/2cbc07c217c30b3f FAIL (2025-12-18 14:28:05 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, Piece ID: O7IY6V6T4UILG3SXQ6K3SPXBVA2VLB22PWVQYIYORGLLYN6LE64A) context deadline exceeded; piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: I2SLWBEKFHZXWYTBZF73DICAZ4ZBXQYU6DES2QWLJMUCFZOCC7CA) context deadline exceeded ======LOGS====== ec 18 14:26:45.953557484 (310h57m45.824640529s) [upload=272678]: Starting upload Dec 18 14:26:45.953631205 (310h57m45.82471425s) [upload=272678]: Done starting upload Dec 18 14:26:45.953727147 (310h57m45.824810202s) [upload=272678]: Uploading segments... Dec 18 14:26:45.953920536 (310h57m45.825003581s) [upload=272678]: Got next segment. Inline: false Dec 18 14:26:45.953955743 (310h57m45.825038798s) [upload=272678]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 18 14:26:45.953963277 (310h57m45.825046322s) [upload=272678]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 18 14:26:46.201381593 (310h57m46.072464628s) [seg_pos={0 0}, upload=272678]: Begin upload segment... Dec 18 14:26:46.201430356 (310h57m46.072513391s) [seg_pos={0 0}, upload=272678]: Done begin upload segment. Dec 18 14:26:46.201523023 (310h57m46.072606088s) [noise=true , address=145.79.227.45:4231 , piece_id=V7B6RPI74SKRJQZCUWFHB2ECD46T3V224VKBL47NIIK5YH6PJCUQ, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.201534584 (310h57m46.072617619s) [noise=true , address=45.91.116.8:4048 , piece_id=BHQYLAJFKTBQ2WGY4XGRI4UOPTGB3SFEUVYZ7XWAC2BKA3ISU3WA, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.203237186 (310h57m46.074320231s) [noise=true , address=45.91.116.14:4190 , piece_id=I2SLWBEKFHZXWYTBZF73DICAZ4ZBXQYU6DES2QWLJMUCFZOCC7CA, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.203669961 (310h57m46.074752996s) [noise=true , address=145.79.227.36:4519 , piece_id=3E54JIZJCR55H36BEOLTDKWRMYA3WHNDPBSALSMNRY45O3BYO6JA, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.203970172 (310h57m46.075053207s) [noise=true , address=145.79.227.38:4978 , piece_id=FRWJ6Z3EH45YRMQVU235FJB75JWK7HOCLZXS7ASITOHLTL3YY5PQ, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.204250205 (310h57m46.07533324s) [noise=true , address=45.91.116.10:4084 , piece_id=UOTQVSUYETMA7UOEURUAT5RPX4QUO4FOXSLQY7UDUKJBCYCOH5BA, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.204534577 (310h57m46.075617612s) [noise=true , address=45.91.116.5:4299 , piece_id=SIZBVTWHVPXBL7OMRRQ467KFLRB56JNA5WDRNSXSNHPSV7UYG5LQ, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.204821995 (310h57m46.07590504s) [noise=true , address=145.79.227.29:4943 , piece_id=4DWWCYWOAU7NSESXDCONC6THXO3TTUYJMN7G6PTG2CP2OOOY2USA, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.206113864 (310h57m46.0771969s) [noise=true , address=77.90.31.11:4556 , piece_id=4GEBIFAQJYJX4KTPHTYUODQZ6ABTTPOA5SYOJBGUV7YDFIHHVO2Q, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.206151877 (310h57m46.077234912s) [noise=true , address=145.79.227.3:4150 , piece_id=XR5E35XW6KBFTREM2GFHARKAAIAINOWMLVYDKKBPFQBXJZOFCC2Q, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.207752204 (310h57m46.078835239s) [noise=true , address=145.79.227.39:4013 , piece_id=2SYWMQU5R7MCTH2E4X732AOVIRVPGUNNXLIN3UFCU7WF4BEJUACQ, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.210366552 (310h57m46.081449597s) [noise=true , address=145.79.227.35:4482 , piece_id=3EGA2CLZBLJWSSULQCBLEEHCMCJDE2FSOY4FCANAQXPEYTWIF53A, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.211613657 (310h57m46.082696692s) [noise=true , address=145.79.227.47:4266 , piece_id=4444BNJTIBUQWLNGTEK3IFD4TIWTCIBCWUHSYBMD3FGZSNDPWT3Q, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.212828561 (310h57m46.083911596s) [noise=true , address=145.79.227.41:4086 , piece_id=DFSNBYWUJ6Z65VTU5ME2IGEDB6KLMTAJM65RC44FUVYHTPEW4ZSA, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.215707031 (310h57m46.086790066s) [noise=true , address=45.91.116.13:4155 , piece_id=O7IY6V6T4UILG3SXQ6K3SPXBVA2VLB22PWVQYIYORGLLYN6LE64A, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.216889753 (310h57m46.087972798s) [noise=true , address=145.79.227.31:4797 , piece_id=F5WHLSRZSSZ4CRJRR2ZPYGGG67SHQ3QMAIHIC42NOVDW2UXARINQ, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:47.912125819 (310h57m47.783208865s) [upload=272678]: Next returned nil segment Dec 18 14:26:47.912143884 (310h57m47.783226929s) [upload=272678]: Waiting for error group managing segments... Dec 18 14:26:48.193212096 (310h57m48.064295131s) [noise=true , address=145.79.227.45:4231 , piece_id=V7B6RPI74SKRJQZCUWFHB2ECD46T3V224VKBL47NIIK5YH6PJCUQ, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.207387668 (310h57m48.078470703s) [noise=true , address=145.79.227.31:4797 , piece_id=F5WHLSRZSSZ4CRJRR2ZPYGGG67SHQ3QMAIHIC42NOVDW2UXARINQ, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.219064714 (310h57m48.090147759s) [noise=true , address=145.79.227.39:4013 , piece_id=2SYWMQU5R7MCTH2E4X732AOVIRVPGUNNXLIN3UFCU7WF4BEJUACQ, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.227887805 (310h57m48.09897085s) [noise=true , address=145.79.227.36:4519 , piece_id=3E54JIZJCR55H36BEOLTDKWRMYA3WHNDPBSALSMNRY45O3BYO6JA, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.229044407 (310h57m48.100127442s) [noise=true , address=145.79.227.41:4086 , piece_id=DFSNBYWUJ6Z65VTU5ME2IGEDB6KLMTAJM65RC44FUVYHTPEW4ZSA, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.234560812 (310h57m48.105643857s) [noise=true , address=145.79.227.47:4266 , piece_id=4444BNJTIBUQWLNGTEK3IFD4TIWTCIBCWUHSYBMD3FGZSNDPWT3Q, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.237048037 (310h57m48.108131072s) [noise=true , address=145.79.227.35:4482 , piece_id=3EGA2CLZBLJWSSULQCBLEEHCMCJDE2FSOY4FCANAQXPEYTWIF53A, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.267241984 (310h57m48.138325019s) [noise=true , address=145.79.227.29:4943 , piece_id=4DWWCYWOAU7NSESXDCONC6THXO3TTUYJMN7G6PTG2CP2OOOY2USA, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.288475576 (310h57m48.159558621s) [noise=true , address=145.79.227.3:4150 , piece_id=XR5E35XW6KBFTREM2GFHARKAAIAINOWMLVYDKKBPFQBXJZOFCC2Q, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.293478393 (310h57m48.164561458s) [noise=true , address=145.79.227.38:4978 , piece_id=FRWJ6Z3EH45YRMQVU235FJB75JWK7HOCLZXS7ASITOHLTL3YY5PQ, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.423759785 (310h57m48.29484282s) [noise=true , address=45.91.116.14:4190 , piece_id=I2SLWBEKFHZXWYTBZF73DICAZ4ZBXQYU6DES2QWLJMUCFZOCC7CA, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.485697347 (310h57m48.356780392s) [noise=true , address=45.91.116.10:4084 , piece_id=UOTQVSUYETMA7UOEURUAT5RPX4QUO4FOXSLQY7UDUKJBCYCOH5BA, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.540540711 (310h57m48.411623756s) [noise=true , address=45.91.116.13:4155 , piece_id=O7IY6V6T4UILG3SXQ6K3SPXBVA2VLB22PWVQYIYORGLLYN6LE64A, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.540575787 (310h57m48.411658832s) [seg_pos={0 0}, upload=272678]: Segment reached optimal threshold of 13 pieces. Dec 18 14:26:48.540831424 (310h57m48.411914459s) [noise=true , address=45.91.116.8:4048 , piece_id=BHQYLAJFKTBQ2WGY4XGRI4UOPTGB3SFEUVYZ7XWAC2BKA3ISU3WA, seg_pos={0 0}, upload=272678]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: write tcp 188.166.225.170:49328->45.91.116.8:4048: use of closed network connection; write tcp 188.166.225.170:49328->45.91.116.8:4048: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49328->45.91.116.8:4048: use of closed network connection Dec 18 14:26:48.540840511 (310h57m48.411923556s) [noise=true , address=45.91.116.8:4048 , piece_id=BHQYLAJFKTBQ2WGY4XGRI4UOPTGB3SFEUVYZ7XWAC2BKA3ISU3WA, seg_pos={0 0}, upload=272678]: Long tail cancellation due to optimal threshold Dec 18 14:26:48.540888052 (310h57m48.411971107s) [noise=true , address=45.91.116.5:4299 , piece_id=SIZBVTWHVPXBL7OMRRQ467KFLRB56JNA5WDRNSXSNHPSV7UYG5LQ, seg_pos={0 0}, upload=272678]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Dec 18 14:26:48.540899453 (310h57m48.411982498s) [noise=true , address=45.91.116.5:4299 , piece_id=SIZBVTWHVPXBL7OMRRQ467KFLRB56JNA5WDRNSXSNHPSV7UYG5LQ, seg_pos={0 0}, upload=272678]: Long tail cancellation due to optimal threshold Dec 18 14:26:48.541046764 (310h57m48.412129809s) [noise=true , address=77.90.31.11:4556 , piece_id=4GEBIFAQJYJX4KTPHTYUODQZ6ABTTPOA5SYOJBGUV7YDFIHHVO2Q, seg_pos={0 0}, upload=272678]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: expected piece hash; context canceled; EOF Dec 18 14:26:48.541055401 (310h57m48.412138436s) [noise=true , address=77.90.31.11:4556 , piece_id=4GEBIFAQJYJX4KTPHTYUODQZ6ABTTPOA5SYOJBGUV7YDFIHHVO2Q, seg_pos={0 0}, upload=272678]: Long tail cancellation due to optimal threshold Dec 18 14:26:48.541086550 (310h57m48.412169595s) [seg_pos={0 0}, upload=272678]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 18 14:26:48.541155190 (310h57m48.412238226s) [upload=272678]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 18 14:26:48.541159900 (310h57m48.412242935s) [upload=272678]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 18 14:26:48.836195919 (310h57m48.707278954s) [upload=272678]: Done uploading segments... Dec 18 14:26:48.836215697 (310h57m48.707298742s) [upload=272678]: Upload finished. err: <nil> Duration: 5.841090204s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 19 out of 60 is failed (threshold=6) [.XXX.....X..X.X..............X.......X.....X.X.XX...XXXXX.XX] duration: 5.001413726s trace: https://jaeger.dev.storj.io/trace/52122bfeccd0cf7e FAIL (2025-12-18 14:27:30 UTC) orbiter warning: context deadline exceeded Duration: 5.001447289s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [......................................X....................X] duration: 30.002557224s trace: https://jaeger.dev.storj.io/trace/2cbc07c217c30b3f FAIL (2025-12-18 14:27:15 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, Piece ID: O7IY6V6T4UILG3SXQ6K3SPXBVA2VLB22PWVQYIYORGLLYN6LE64A) context deadline exceeded; piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: I2SLWBEKFHZXWYTBZF73DICAZ4ZBXQYU6DES2QWLJMUCFZOCC7CA) context deadline exceeded ======LOGS====== ec 18 14:26:45.953557484 (310h57m45.824640529s) [upload=272678]: Starting upload Dec 18 14:26:45.953631205 (310h57m45.82471425s) [upload=272678]: Done starting upload Dec 18 14:26:45.953727147 (310h57m45.824810202s) [upload=272678]: Uploading segments... Dec 18 14:26:45.953920536 (310h57m45.825003581s) [upload=272678]: Got next segment. Inline: false Dec 18 14:26:45.953955743 (310h57m45.825038798s) [upload=272678]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 18 14:26:45.953963277 (310h57m45.825046322s) [upload=272678]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 18 14:26:46.201381593 (310h57m46.072464628s) [seg_pos={0 0}, upload=272678]: Begin upload segment... Dec 18 14:26:46.201430356 (310h57m46.072513391s) [seg_pos={0 0}, upload=272678]: Done begin upload segment. Dec 18 14:26:46.201523023 (310h57m46.072606088s) [noise=true , address=145.79.227.45:4231 , piece_id=V7B6RPI74SKRJQZCUWFHB2ECD46T3V224VKBL47NIIK5YH6PJCUQ, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.201534584 (310h57m46.072617619s) [noise=true , address=45.91.116.8:4048 , piece_id=BHQYLAJFKTBQ2WGY4XGRI4UOPTGB3SFEUVYZ7XWAC2BKA3ISU3WA, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.203237186 (310h57m46.074320231s) [noise=true , address=45.91.116.14:4190 , piece_id=I2SLWBEKFHZXWYTBZF73DICAZ4ZBXQYU6DES2QWLJMUCFZOCC7CA, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.203669961 (310h57m46.074752996s) [noise=true , address=145.79.227.36:4519 , piece_id=3E54JIZJCR55H36BEOLTDKWRMYA3WHNDPBSALSMNRY45O3BYO6JA, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.203970172 (310h57m46.075053207s) [noise=true , address=145.79.227.38:4978 , piece_id=FRWJ6Z3EH45YRMQVU235FJB75JWK7HOCLZXS7ASITOHLTL3YY5PQ, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.204250205 (310h57m46.07533324s) [noise=true , address=45.91.116.10:4084 , piece_id=UOTQVSUYETMA7UOEURUAT5RPX4QUO4FOXSLQY7UDUKJBCYCOH5BA, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.204534577 (310h57m46.075617612s) [noise=true , address=45.91.116.5:4299 , piece_id=SIZBVTWHVPXBL7OMRRQ467KFLRB56JNA5WDRNSXSNHPSV7UYG5LQ, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.204821995 (310h57m46.07590504s) [noise=true , address=145.79.227.29:4943 , piece_id=4DWWCYWOAU7NSESXDCONC6THXO3TTUYJMN7G6PTG2CP2OOOY2USA, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.206113864 (310h57m46.0771969s) [noise=true , address=77.90.31.11:4556 , piece_id=4GEBIFAQJYJX4KTPHTYUODQZ6ABTTPOA5SYOJBGUV7YDFIHHVO2Q, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.206151877 (310h57m46.077234912s) [noise=true , address=145.79.227.3:4150 , piece_id=XR5E35XW6KBFTREM2GFHARKAAIAINOWMLVYDKKBPFQBXJZOFCC2Q, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.207752204 (310h57m46.078835239s) [noise=true , address=145.79.227.39:4013 , piece_id=2SYWMQU5R7MCTH2E4X732AOVIRVPGUNNXLIN3UFCU7WF4BEJUACQ, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.210366552 (310h57m46.081449597s) [noise=true , address=145.79.227.35:4482 , piece_id=3EGA2CLZBLJWSSULQCBLEEHCMCJDE2FSOY4FCANAQXPEYTWIF53A, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.211613657 (310h57m46.082696692s) [noise=true , address=145.79.227.47:4266 , piece_id=4444BNJTIBUQWLNGTEK3IFD4TIWTCIBCWUHSYBMD3FGZSNDPWT3Q, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.212828561 (310h57m46.083911596s) [noise=true , address=145.79.227.41:4086 , piece_id=DFSNBYWUJ6Z65VTU5ME2IGEDB6KLMTAJM65RC44FUVYHTPEW4ZSA, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.215707031 (310h57m46.086790066s) [noise=true , address=45.91.116.13:4155 , piece_id=O7IY6V6T4UILG3SXQ6K3SPXBVA2VLB22PWVQYIYORGLLYN6LE64A, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:46.216889753 (310h57m46.087972798s) [noise=true , address=145.79.227.31:4797 , piece_id=F5WHLSRZSSZ4CRJRR2ZPYGGG67SHQ3QMAIHIC42NOVDW2UXARINQ, seg_pos={0 0}, upload=272678]: Uploading piece... Dec 18 14:26:47.912125819 (310h57m47.783208865s) [upload=272678]: Next returned nil segment Dec 18 14:26:47.912143884 (310h57m47.783226929s) [upload=272678]: Waiting for error group managing segments... Dec 18 14:26:48.193212096 (310h57m48.064295131s) [noise=true , address=145.79.227.45:4231 , piece_id=V7B6RPI74SKRJQZCUWFHB2ECD46T3V224VKBL47NIIK5YH6PJCUQ, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.207387668 (310h57m48.078470703s) [noise=true , address=145.79.227.31:4797 , piece_id=F5WHLSRZSSZ4CRJRR2ZPYGGG67SHQ3QMAIHIC42NOVDW2UXARINQ, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.219064714 (310h57m48.090147759s) [noise=true , address=145.79.227.39:4013 , piece_id=2SYWMQU5R7MCTH2E4X732AOVIRVPGUNNXLIN3UFCU7WF4BEJUACQ, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.227887805 (310h57m48.09897085s) [noise=true , address=145.79.227.36:4519 , piece_id=3E54JIZJCR55H36BEOLTDKWRMYA3WHNDPBSALSMNRY45O3BYO6JA, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.229044407 (310h57m48.100127442s) [noise=true , address=145.79.227.41:4086 , piece_id=DFSNBYWUJ6Z65VTU5ME2IGEDB6KLMTAJM65RC44FUVYHTPEW4ZSA, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.234560812 (310h57m48.105643857s) [noise=true , address=145.79.227.47:4266 , piece_id=4444BNJTIBUQWLNGTEK3IFD4TIWTCIBCWUHSYBMD3FGZSNDPWT3Q, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.237048037 (310h57m48.108131072s) [noise=true , address=145.79.227.35:4482 , piece_id=3EGA2CLZBLJWSSULQCBLEEHCMCJDE2FSOY4FCANAQXPEYTWIF53A, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.267241984 (310h57m48.138325019s) [noise=true , address=145.79.227.29:4943 , piece_id=4DWWCYWOAU7NSESXDCONC6THXO3TTUYJMN7G6PTG2CP2OOOY2USA, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.288475576 (310h57m48.159558621s) [noise=true , address=145.79.227.3:4150 , piece_id=XR5E35XW6KBFTREM2GFHARKAAIAINOWMLVYDKKBPFQBXJZOFCC2Q, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.293478393 (310h57m48.164561458s) [noise=true , address=145.79.227.38:4978 , piece_id=FRWJ6Z3EH45YRMQVU235FJB75JWK7HOCLZXS7ASITOHLTL3YY5PQ, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.423759785 (310h57m48.29484282s) [noise=true , address=45.91.116.14:4190 , piece_id=I2SLWBEKFHZXWYTBZF73DICAZ4ZBXQYU6DES2QWLJMUCFZOCC7CA, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.485697347 (310h57m48.356780392s) [noise=true , address=45.91.116.10:4084 , piece_id=UOTQVSUYETMA7UOEURUAT5RPX4QUO4FOXSLQY7UDUKJBCYCOH5BA, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.540540711 (310h57m48.411623756s) [noise=true , address=45.91.116.13:4155 , piece_id=O7IY6V6T4UILG3SXQ6K3SPXBVA2VLB22PWVQYIYORGLLYN6LE64A, seg_pos={0 0}, upload=272678]: Done uploading piece. err: <nil> Dec 18 14:26:48.540575787 (310h57m48.411658832s) [seg_pos={0 0}, upload=272678]: Segment reached optimal threshold of 13 pieces. Dec 18 14:26:48.540831424 (310h57m48.411914459s) [noise=true , address=45.91.116.8:4048 , piece_id=BHQYLAJFKTBQ2WGY4XGRI4UOPTGB3SFEUVYZ7XWAC2BKA3ISU3WA, seg_pos={0 0}, upload=272678]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: write tcp 188.166.225.170:49328->45.91.116.8:4048: use of closed network connection; write tcp 188.166.225.170:49328->45.91.116.8:4048: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49328->45.91.116.8:4048: use of closed network connection Dec 18 14:26:48.540840511 (310h57m48.411923556s) [noise=true , address=45.91.116.8:4048 , piece_id=BHQYLAJFKTBQ2WGY4XGRI4UOPTGB3SFEUVYZ7XWAC2BKA3ISU3WA, seg_pos={0 0}, upload=272678]: Long tail cancellation due to optimal threshold Dec 18 14:26:48.540888052 (310h57m48.411971107s) [noise=true , address=45.91.116.5:4299 , piece_id=SIZBVTWHVPXBL7OMRRQ467KFLRB56JNA5WDRNSXSNHPSV7UYG5LQ, seg_pos={0 0}, upload=272678]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Dec 18 14:26:48.540899453 (310h57m48.411982498s) [noise=true , address=45.91.116.5:4299 , piece_id=SIZBVTWHVPXBL7OMRRQ467KFLRB56JNA5WDRNSXSNHPSV7UYG5LQ, seg_pos={0 0}, upload=272678]: Long tail cancellation due to optimal threshold Dec 18 14:26:48.541046764 (310h57m48.412129809s) [noise=true , address=77.90.31.11:4556 , piece_id=4GEBIFAQJYJX4KTPHTYUODQZ6ABTTPOA5SYOJBGUV7YDFIHHVO2Q, seg_pos={0 0}, upload=272678]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: expected piece hash; context canceled; EOF Dec 18 14:26:48.541055401 (310h57m48.412138436s) [noise=true , address=77.90.31.11:4556 , piece_id=4GEBIFAQJYJX4KTPHTYUODQZ6ABTTPOA5SYOJBGUV7YDFIHHVO2Q, seg_pos={0 0}, upload=272678]: Long tail cancellation due to optimal threshold Dec 18 14:26:48.541086550 (310h57m48.412169595s) [seg_pos={0 0}, upload=272678]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 18 14:26:48.541155190 (310h57m48.412238226s) [upload=272678]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 18 14:26:48.541159900 (310h57m48.412242935s) [upload=272678]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 18 14:26:48.836195919 (310h57m48.707278954s) [upload=272678]: Done uploading segments... Dec 18 14:26:48.836215697 (310h57m48.707298742s) [upload=272678]: Upload finished. err: <nil> Duration: 30.002582582s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 18 out of 60 is failed (threshold=6) [..XXX.....X..X.X..............X.......X.....X.X.XX...XXXXX.X] duration: 5.001069543s trace: https://jaeger.dev.storj.io/trace/4afef2eeff4c0bc1 FAIL (2025-12-18 14:26:40 UTC) orbiter warning: context deadline exceeded Duration: 5.001092207s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 5.927835241s trace: https://jaeger.dev.storj.io/trace/16860fde35486b5b FAIL (2025-12-18 14:25:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 18 14:10:58.398752236 (310h41m58.269835281s) [upload=272455]: Starting upload Dec 18 14:10:58.398827229 (310h41m58.269910264s) [upload=272455]: Done starting upload Dec 18 14:10:58.398962777 (310h41m58.270045812s) [upload=272455]: Uploading segments... Dec 18 14:10:58.399055643 (310h41m58.270138678s) [upload=272455]: Got next segment. Inline: false Dec 18 14:10:58.399103244 (310h41m58.270186279s) [upload=272455]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 18 14:10:58.399110608 (310h41m58.270193643s) [upload=272455]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 18 14:10:58.658153258 (310h41m58.529236303s) [seg_pos={0 0}, upload=272455]: Begin upload segment... Dec 18 14:10:58.658215857 (310h41m58.529298892s) [seg_pos={0 0}, upload=272455]: Done begin upload segment. Dec 18 14:10:58.658255322 (310h41m58.529338367s) [noise=true , address=145.79.227.43:4158 , piece_id=YFPLBK67MJMDPNCYRQECZNTFJNVE2K5X3IVXWLZP5NE75ZV6MB4A, seg_pos={0 0}, upload=272455]: Uploading piece... Dec 18 14:10:58.658535004 (310h41m58.52961805s) [noise=true , address=145.79.227.41:4087 , piece_id=CTM6GC2THQRGIHRQYQRZ5B4X7LJFZK5SRZ3B5WUVMBLXX3H5SO2A, seg_pos={0 0}, upload=272455]: Uploading piece... Dec 18 14:10:58.659508749 (310h41m58.530591784s) [noise=true , address=145.79.227.3:4150 , piece_id=LIEYC6BCTKMQYJ7NSYO4FOB2Z76KAIKAFFV75KBSX26P2MQ7PLSQ, seg_pos={0 0}, upload=272455]: Uploading piece... Dec 18 14:10:58.659943366 (310h41m58.531026411s) [noise=true , address=145.79.227.21:4473 , piece_id=GPWWFTRAFVH2YTSFV2BKNXU7RZ4FRVJHQSN72SLEHQ6FZMJDM37Q, seg_pos={0 0}, upload=272455]: Uploading piece... Dec 18 14:10:58.660240492 (310h41m58.531323537s) [noise=true , address=145.79.227.38:4979 , piece_id=OH46FVYFB6XP4UP423HF4PWPIO5FD6CNKXXS4VGNAQX7HDNFOGKA, seg_pos={0 0}, upload=272455]: Uploading piece... Dec 18 14:10:58.660540573 (310h41m58.531623618s) [noise=true , address=77.90.31.12:4590 , piece_id=2BTRJAJCSCZY5DDLRWCA2MNBDKIK4N2MKVVBJMWZWU3HCMKZY7VQ, seg_pos={0 0}, upload=272455]: Uploading piece... Dec 18 14:10:58.661050785 (310h41m58.53213383s) [noise=true , address=145.79.227.36:4517 , piece_id=2XK7T4CUDULYM45PUJ63D5EFGCMYWHW3TSPDUHPKSNHF4N5HXONA, seg_pos={0 0}, upload=272455]: Uploading piece... Dec 18 14:10:58.662869197 (310h41m58.533952242s) [noise=true , address=145.79.227.51:4409 , piece_id=W5LIBK6QBYHILK7Y5325QYF5TFQ2IEEVXSWLNNKZZDWE65U67LIQ, seg_pos={0 0}, upload=272455]: Uploading piece... Dec 18 14:10:58.663106028 (310h41m58.534189073s) [noise=true , address=145.79.227.52:4447 , piece_id=GGIDHSTBE5IQJK5W7OVAU4BV3T6SN56IDWTL45RWTUIFNINGW5PA, seg_pos={0 0}, upload=272455]: Uploading piece... Dec 18 14:10:58.664904303 (310h41m58.535987348s) [noise=true , address=145.79.227.37:4565 , piece_id=REQ4DLQLJEKOO4Q5YFVTV5R456TMCDVYW2RFJDVUO7KZZL3RZT5A, seg_pos={0 0}, upload=272455]: Uploading piece... Dec 18 14:10:58.666331190 (310h41m58.537414225s) [noise=true , address=45.91.116.14:4190 , piece_id=QMPTQA3MBIQ5PDW5JPA75FUSJ6EEXES27TQ3DJVG5RUW4SHJNLXA, seg_pos={0 0}, upload=272455]: Uploading piece... Dec 18 14:10:58.667746544 (310h41m58.538829589s) [noise=true , address=45.91.116.8:4046 , piece_id=GRCI6RP63B7IKPVXCMDMFQNWCNJJ4CMZZFHALC73QVDKTW2Q6C2A, seg_pos={0 0}, upload=272455]: Uploading piece... Dec 18 14:10:58.669039206 (310h41m58.540122251s) [noise=true , address=77.90.31.14:4661 , piece_id=Z4IQ4TJKA6IRY2SSYJ6BBNEBCVXGOXRLMDYRABA5MQ7OP5V3UEPQ, seg_pos={0 0}, upload=272455]: Uploading piece... Dec 18 14:10:58.670492172 (310h41m58.541575218s) [noise=true , address=45.91.116.9:4264 , piece_id=BWC5LRVPTOEYHTM77SZV53TV2PQIWRVLYPOJEDKIGZLUAPTY4TYQ, seg_pos={0 0}, upload=272455]: Uploading piece... Dec 18 14:10:58.671601074 (310h41m58.542684119s) [noise=true , address=145.79.227.45:4230 , piece_id=TK2HOWL5IUH2U3Z4EWMUXMFCU4TH7C32MALBKU3HBLR2Y4DYDPFA, seg_pos={0 0}, upload=272455]: Uploading piece... Dec 18 14:10:58.672764109 (310h41m58.543847154s) [noise=true , address=145.79.227.50:4374 , piece_id=FOC326BQVXZUI62ZY5VSJ7M54QWLWRVCTEQ65CLLM43OKXESATTQ, seg_pos={0 0}, upload=272455]: Uploading piece... Dec 18 14:11:00.261349048 (310h42m0.132432093s) [upload=272455]: Next returned nil segment Dec 18 14:11:00.261365791 (310h42m0.132448836s) [upload=272455]: Waiting for error group managing segments... Dec 18 14:11:00.646239579 (310h42m0.517322614s) [noise=true , address=145.79.227.41:4087 , piece_id=CTM6GC2THQRGIHRQYQRZ5B4X7LJFZK5SRZ3B5WUVMBLXX3H5SO2A, seg_pos={0 0}, upload=272455]: Done uploading piece. err: <nil> Dec 18 14:11:00.647832222 (310h42m0.518915267s) [noise=true , address=145.79.227.38:4979 , piece_id=OH46FVYFB6XP4UP423HF4PWPIO5FD6CNKXXS4VGNAQX7HDNFOGKA, seg_pos={0 0}, upload=272455]: Done uploading piece. err: <nil> Dec 18 14:11:00.653396516 (310h42m0.524479551s) [noise=true , address=145.79.227.43:4158 , piece_id=YFPLBK67MJMDPNCYRQECZNTFJNVE2K5X3IVXWLZP5NE75ZV6MB4A, seg_pos={0 0}, upload=272455]: Done uploading piece. err: <nil> Dec 18 14:11:00.659446878 (310h42m0.530529923s) [noise=true , address=145.79.227.36:4517 , piece_id=2XK7T4CUDULYM45PUJ63D5EFGCMYWHW3TSPDUHPKSNHF4N5HXONA, seg_pos={0 0}, upload=272455]: Done uploading piece. err: <nil> Dec 18 14:11:00.668013681 (310h42m0.539096726s) [noise=true , address=145.79.227.37:4565 , piece_id=REQ4DLQLJEKOO4Q5YFVTV5R456TMCDVYW2RFJDVUO7KZZL3RZT5A, seg_pos={0 0}, upload=272455]: Done uploading piece. err: <nil> Dec 18 14:11:00.676606744 (310h42m0.547689789s) [noise=true , address=145.79.227.52:4447 , piece_id=GGIDHSTBE5IQJK5W7OVAU4BV3T6SN56IDWTL45RWTUIFNINGW5PA, seg_pos={0 0}, upload=272455]: Done uploading piece. err: <nil> Dec 18 14:11:00.734809955 (310h42m0.605893s) [noise=true , address=145.79.227.51:4409 , piece_id=W5LIBK6QBYHILK7Y5325QYF5TFQ2IEEVXSWLNNKZZDWE65U67LIQ, seg_pos={0 0}, upload=272455]: Done uploading piece. err: <nil> Dec 18 14:11:00.736405162 (310h42m0.607488207s) [noise=true , address=145.79.227.50:4374 , piece_id=FOC326BQVXZUI62ZY5VSJ7M54QWLWRVCTEQ65CLLM43OKXESATTQ, seg_pos={0 0}, upload=272455]: Done uploading piece. err: <nil> Dec 18 14:11:00.739609083 (310h42m0.610692128s) [noise=true , address=145.79.227.45:4230 , piece_id=TK2HOWL5IUH2U3Z4EWMUXMFCU4TH7C32MALBKU3HBLR2Y4DYDPFA, seg_pos={0 0}, upload=272455]: Done uploading piece. err: <nil> Dec 18 14:11:00.750715051 (310h42m0.621798086s) [noise=true , address=145.79.227.21:4473 , piece_id=GPWWFTRAFVH2YTSFV2BKNXU7RZ4FRVJHQSN72SLEHQ6FZMJDM37Q, seg_pos={0 0}, upload=272455]: Done uploading piece. err: <nil> Dec 18 14:11:00.753496537 (310h42m0.624579582s) [noise=true , address=145.79.227.3:4150 , piece_id=LIEYC6BCTKMQYJ7NSYO4FOB2Z76KAIKAFFV75KBSX26P2MQ7PLSQ, seg_pos={0 0}, upload=272455]: Done uploading piece. err: <nil> Dec 18 14:11:00.841269074 (310h42m0.712352109s) [noise=true , address=77.90.31.12:4590 , piece_id=2BTRJAJCSCZY5DDLRWCA2MNBDKIK4N2MKVVBJMWZWU3HCMKZY7VQ, seg_pos={0 0}, upload=272455]: Done uploading piece. err: <nil> Dec 18 14:11:00.892845626 (310h42m0.763928661s) [noise=true , address=45.91.116.8:4046 , piece_id=GRCI6RP63B7IKPVXCMDMFQNWCNJJ4CMZZFHALC73QVDKTW2Q6C2A, seg_pos={0 0}, upload=272455]: Done uploading piece. err: <nil> Dec 18 14:11:00.892865524 (310h42m0.763948559s) [seg_pos={0 0}, upload=272455]: Segment reached optimal threshold of 13 pieces. Dec 18 14:11:00.893020539 (310h42m0.764103584s) [noise=true , address=45.91.116.9:4264 , piece_id=BWC5LRVPTOEYHTM77SZV53TV2PQIWRVLYPOJEDKIGZLUAPTY4TYQ, seg_pos={0 0}, upload=272455]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: expected piece hash; context canceled; EOF Dec 18 14:11:00.893027102 (310h42m0.764110147s) [noise=true , address=45.91.116.9:4264 , piece_id=BWC5LRVPTOEYHTM77SZV53TV2PQIWRVLYPOJEDKIGZLUAPTY4TYQ, seg_pos={0 0}, upload=272455]: Long tail cancellation due to optimal threshold Dec 18 14:11:00.893274413 (310h42m0.764357458s) [noise=true , address=45.91.116.14:4190 , piece_id=QMPTQA3MBIQ5PDW5JPA75FUSJ6EEXES27TQ3DJVG5RUW4SHJNLXA, seg_pos={0 0}, upload=272455]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: expected piece hash; context canceled; EOF Dec 18 14:11:00.893281326 (310h42m0.764364371s) [noise=true , address=45.91.116.14:4190 , piece_id=QMPTQA3MBIQ5PDW5JPA75FUSJ6EEXES27TQ3DJVG5RUW4SHJNLXA, seg_pos={0 0}, upload=272455]: Long tail cancellation due to optimal threshold Dec 18 14:11:00.893378692 (310h42m0.764461737s) [noise=true , address=77.90.31.14:4661 , piece_id=Z4IQ4TJKA6IRY2SSYJ6BBNEBCVXGOXRLMDYRABA5MQ7OP5V3UEPQ, seg_pos={0 0}, upload=272455]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: expected piece hash; context canceled; EOF Dec 18 14:11:00.893384182 (310h42m0.764467227s) [noise=true , address=77.90.31.14:4661 , piece_id=Z4IQ4TJKA6IRY2SSYJ6BBNEBCVXGOXRLMDYRABA5MQ7OP5V3UEPQ, seg_pos={0 0}, upload=272455]: Long tail cancellation due to optimal threshold Dec 18 14:11:00.893398469 (310h42m0.764481504s) [seg_pos={0 0}, upload=272455]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 18 14:11:00.893460958 (310h42m0.764544003s) [upload=272455]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 18 14:11:00.893465627 (310h42m0.764548672s) [upload=272455]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 18 14:11:01.215063464 (310h42m1.086146509s) [upload=272455]: Done uploading segments... Dec 18 14:11:01.215089965 (310h42m1.08617301s) [upload=272455]: Upload finished. err: <nil> Duration: 5.927860047s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 17 out of 60 is failed (threshold=6) [...XXX.....X..X.X..............X.......X.....X.X.XX...XXXXX.] duration: 3.775710444s trace: https://jaeger.dev.storj.io/trace/4e826141d6998096 FAIL (2025-12-18 14:25:37 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.775859408s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 5.785213833s trace: https://jaeger.dev.storj.io/trace/16860fde35486b5b FAIL (2025-12-18 14:25:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 18 14:10:58.398752236 (310h41m58.269835281s) [upload=272455]: Starting upload Dec 18 14:10:58.398827229 (310h41m58.269910264s) [upload=272455]: Done starting upload Dec 18 14:10:58.398962777 (310h41m58.270045812s) [upload=272455]: Uploading segments... Dec 18 14:10:58.399055643 (310h41m58.270138678s) [upload=272455]: Got next segment. Inline: false Dec 18 14:10:58.399103244 (310h41m58.270186279s) [upload=272455]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 18 14:10:58.399110608 (310h41m58.270193643s) [upload=272455]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 18 14:10:58.658153258 (310h41m58.529236303s) [seg_pos={0 0}, upload=272455]: Begin upload segment... Dec 18 14:10:58.658215857 (310h41m58.529298892s) [seg_pos={0 0}, upload=272455]: Done begin upload segment. Dec 18 14:10:58.658255322 (310h41m58.529338367s) [noise=true , address=145.79.227.43:4158 , piece_id=YFPLBK67MJMDPNCYRQECZNTFJNVE2K5X3IVXWLZP5NE75ZV6MB4A, seg_pos={0 0}, upload=272455]: Uploading piece... Dec 18 14:10:58.658535004 (310h41m58.52961805s) [noise=true , address=145.79.227.41:4087 , piece_id=CTM6GC2THQRGIHRQYQRZ5B4X7LJFZK5SRZ3B5WUVMBLXX3H5SO2A, seg_pos={0 0}, upload=272455]: Uploading piece... Dec 18 14:10:58.659508749 (310h41m58.530591784s) [noise=true , address=145.79.227.3:4150 , piece_id=LIEYC6BCTKMQYJ7NSYO4FOB2Z76KAIKAFFV75KBSX26P2MQ7PLSQ, seg_pos={0 0}, upload=272455]: Uploading piece... Dec 18 14:10:58.659943366 (310h41m58.531026411s) [noise=true , address=145.79.227.21:4473 , piece_id=GPWWFTRAFVH2YTSFV2BKNXU7RZ4FRVJHQSN72SLEHQ6FZMJDM37Q, seg_pos={0 0}, upload=272455]: Uploading piece... Dec 18 14:10:58.660240492 (310h41m58.531323537s) [noise=true , address=145.79.227.38:4979 , piece_id=OH46FVYFB6XP4UP423HF4PWPIO5FD6CNKXXS4VGNAQX7HDNFOGKA, seg_pos={0 0}, upload=272455]: Uploading piece... Dec 18 14:10:58.660540573 (310h41m58.531623618s) [noise=true , address=77.90.31.12:4590 , piece_id=2BTRJAJCSCZY5DDLRWCA2MNBDKIK4N2MKVVBJMWZWU3HCMKZY7VQ, seg_pos={0 0}, upload=272455]: Uploading piece... Dec 18 14:10:58.661050785 (310h41m58.53213383s) [noise=true , address=145.79.227.36:4517 , piece_id=2XK7T4CUDULYM45PUJ63D5EFGCMYWHW3TSPDUHPKSNHF4N5HXONA, seg_pos={0 0}, upload=272455]: Uploading piece... Dec 18 14:10:58.662869197 (310h41m58.533952242s) [noise=true , address=145.79.227.51:4409 , piece_id=W5LIBK6QBYHILK7Y5325QYF5TFQ2IEEVXSWLNNKZZDWE65U67LIQ, seg_pos={0 0}, upload=272455]: Uploading piece... Dec 18 14:10:58.663106028 (310h41m58.534189073s) [noise=true , address=145.79.227.52:4447 , piece_id=GGIDHSTBE5IQJK5W7OVAU4BV3T6SN56IDWTL45RWTUIFNINGW5PA, seg_pos={0 0}, upload=272455]: Uploading piece... Dec 18 14:10:58.664904303 (310h41m58.535987348s) [noise=true , address=145.79.227.37:4565 , piece_id=REQ4DLQLJEKOO4Q5YFVTV5R456TMCDVYW2RFJDVUO7KZZL3RZT5A, seg_pos={0 0}, upload=272455]: Uploading piece... Dec 18 14:10:58.666331190 (310h41m58.537414225s) [noise=true , address=45.91.116.14:4190 , piece_id=QMPTQA3MBIQ5PDW5JPA75FUSJ6EEXES27TQ3DJVG5RUW4SHJNLXA, seg_pos={0 0}, upload=272455]: Uploading piece... Dec 18 14:10:58.667746544 (310h41m58.538829589s) [noise=true , address=45.91.116.8:4046 , piece_id=GRCI6RP63B7IKPVXCMDMFQNWCNJJ4CMZZFHALC73QVDKTW2Q6C2A, seg_pos={0 0}, upload=272455]: Uploading piece... Dec 18 14:10:58.669039206 (310h41m58.540122251s) [noise=true , address=77.90.31.14:4661 , piece_id=Z4IQ4TJKA6IRY2SSYJ6BBNEBCVXGOXRLMDYRABA5MQ7OP5V3UEPQ, seg_pos={0 0}, upload=272455]: Uploading piece... Dec 18 14:10:58.670492172 (310h41m58.541575218s) [noise=true , address=45.91.116.9:4264 , piece_id=BWC5LRVPTOEYHTM77SZV53TV2PQIWRVLYPOJEDKIGZLUAPTY4TYQ, seg_pos={0 0}, upload=272455]: Uploading piece... Dec 18 14:10:58.671601074 (310h41m58.542684119s) [noise=true , address=145.79.227.45:4230 , piece_id=TK2HOWL5IUH2U3Z4EWMUXMFCU4TH7C32MALBKU3HBLR2Y4DYDPFA, seg_pos={0 0}, upload=272455]: Uploading piece... Dec 18 14:10:58.672764109 (310h41m58.543847154s) [noise=true , address=145.79.227.50:4374 , piece_id=FOC326BQVXZUI62ZY5VSJ7M54QWLWRVCTEQ65CLLM43OKXESATTQ, seg_pos={0 0}, upload=272455]: Uploading piece... Dec 18 14:11:00.261349048 (310h42m0.132432093s) [upload=272455]: Next returned nil segment Dec 18 14:11:00.261365791 (310h42m0.132448836s) [upload=272455]: Waiting for error group managing segments... Dec 18 14:11:00.646239579 (310h42m0.517322614s) [noise=true , address=145.79.227.41:4087 , piece_id=CTM6GC2THQRGIHRQYQRZ5B4X7LJFZK5SRZ3B5WUVMBLXX3H5SO2A, seg_pos={0 0}, upload=272455]: Done uploading piece. err: <nil> Dec 18 14:11:00.647832222 (310h42m0.518915267s) [noise=true , address=145.79.227.38:4979 , piece_id=OH46FVYFB6XP4UP423HF4PWPIO5FD6CNKXXS4VGNAQX7HDNFOGKA, seg_pos={0 0}, upload=272455]: Done uploading piece. err: <nil> Dec 18 14:11:00.653396516 (310h42m0.524479551s) [noise=true , address=145.79.227.43:4158 , piece_id=YFPLBK67MJMDPNCYRQECZNTFJNVE2K5X3IVXWLZP5NE75ZV6MB4A, seg_pos={0 0}, upload=272455]: Done uploading piece. err: <nil> Dec 18 14:11:00.659446878 (310h42m0.530529923s) [noise=true , address=145.79.227.36:4517 , piece_id=2XK7T4CUDULYM45PUJ63D5EFGCMYWHW3TSPDUHPKSNHF4N5HXONA, seg_pos={0 0}, upload=272455]: Done uploading piece. err: <nil> Dec 18 14:11:00.668013681 (310h42m0.539096726s) [noise=true , address=145.79.227.37:4565 , piece_id=REQ4DLQLJEKOO4Q5YFVTV5R456TMCDVYW2RFJDVUO7KZZL3RZT5A, seg_pos={0 0}, upload=272455]: Done uploading piece. err: <nil> Dec 18 14:11:00.676606744 (310h42m0.547689789s) [noise=true , address=145.79.227.52:4447 , piece_id=GGIDHSTBE5IQJK5W7OVAU4BV3T6SN56IDWTL45RWTUIFNINGW5PA, seg_pos={0 0}, upload=272455]: Done uploading piece. err: <nil> Dec 18 14:11:00.734809955 (310h42m0.605893s) [noise=true , address=145.79.227.51:4409 , piece_id=W5LIBK6QBYHILK7Y5325QYF5TFQ2IEEVXSWLNNKZZDWE65U67LIQ, seg_pos={0 0}, upload=272455]: Done uploading piece. err: <nil> Dec 18 14:11:00.736405162 (310h42m0.607488207s) [noise=true , address=145.79.227.50:4374 , piece_id=FOC326BQVXZUI62ZY5VSJ7M54QWLWRVCTEQ65CLLM43OKXESATTQ, seg_pos={0 0}, upload=272455]: Done uploading piece. err: <nil> Dec 18 14:11:00.739609083 (310h42m0.610692128s) [noise=true , address=145.79.227.45:4230 , piece_id=TK2HOWL5IUH2U3Z4EWMUXMFCU4TH7C32MALBKU3HBLR2Y4DYDPFA, seg_pos={0 0}, upload=272455]: Done uploading piece. err: <nil> Dec 18 14:11:00.750715051 (310h42m0.621798086s) [noise=true , address=145.79.227.21:4473 , piece_id=GPWWFTRAFVH2YTSFV2BKNXU7RZ4FRVJHQSN72SLEHQ6FZMJDM37Q, seg_pos={0 0}, upload=272455]: Done uploading piece. err: <nil> Dec 18 14:11:00.753496537 (310h42m0.624579582s) [noise=true , address=145.79.227.3:4150 , piece_id=LIEYC6BCTKMQYJ7NSYO4FOB2Z76KAIKAFFV75KBSX26P2MQ7PLSQ, seg_pos={0 0}, upload=272455]: Done uploading piece. err: <nil> Dec 18 14:11:00.841269074 (310h42m0.712352109s) [noise=true , address=77.90.31.12:4590 , piece_id=2BTRJAJCSCZY5DDLRWCA2MNBDKIK4N2MKVVBJMWZWU3HCMKZY7VQ, seg_pos={0 0}, upload=272455]: Done uploading piece. err: <nil> Dec 18 14:11:00.892845626 (310h42m0.763928661s) [noise=true , address=45.91.116.8:4046 , piece_id=GRCI6RP63B7IKPVXCMDMFQNWCNJJ4CMZZFHALC73QVDKTW2Q6C2A, seg_pos={0 0}, upload=272455]: Done uploading piece. err: <nil> Dec 18 14:11:00.892865524 (310h42m0.763948559s) [seg_pos={0 0}, upload=272455]: Segment reached optimal threshold of 13 pieces. Dec 18 14:11:00.893020539 (310h42m0.764103584s) [noise=true , address=45.91.116.9:4264 , piece_id=BWC5LRVPTOEYHTM77SZV53TV2PQIWRVLYPOJEDKIGZLUAPTY4TYQ, seg_pos={0 0}, upload=272455]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: expected piece hash; context canceled; EOF Dec 18 14:11:00.893027102 (310h42m0.764110147s) [noise=true , address=45.91.116.9:4264 , piece_id=BWC5LRVPTOEYHTM77SZV53TV2PQIWRVLYPOJEDKIGZLUAPTY4TYQ, seg_pos={0 0}, upload=272455]: Long tail cancellation due to optimal threshold Dec 18 14:11:00.893274413 (310h42m0.764357458s) [noise=true , address=45.91.116.14:4190 , piece_id=QMPTQA3MBIQ5PDW5JPA75FUSJ6EEXES27TQ3DJVG5RUW4SHJNLXA, seg_pos={0 0}, upload=272455]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: expected piece hash; context canceled; EOF Dec 18 14:11:00.893281326 (310h42m0.764364371s) [noise=true , address=45.91.116.14:4190 , piece_id=QMPTQA3MBIQ5PDW5JPA75FUSJ6EEXES27TQ3DJVG5RUW4SHJNLXA, seg_pos={0 0}, upload=272455]: Long tail cancellation due to optimal threshold Dec 18 14:11:00.893378692 (310h42m0.764461737s) [noise=true , address=77.90.31.14:4661 , piece_id=Z4IQ4TJKA6IRY2SSYJ6BBNEBCVXGOXRLMDYRABA5MQ7OP5V3UEPQ, seg_pos={0 0}, upload=272455]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: expected piece hash; context canceled; EOF Dec 18 14:11:00.893384182 (310h42m0.764467227s) [noise=true , address=77.90.31.14:4661 , piece_id=Z4IQ4TJKA6IRY2SSYJ6BBNEBCVXGOXRLMDYRABA5MQ7OP5V3UEPQ, seg_pos={0 0}, upload=272455]: Long tail cancellation due to optimal threshold Dec 18 14:11:00.893398469 (310h42m0.764481504s) [seg_pos={0 0}, upload=272455]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 18 14:11:00.893460958 (310h42m0.764544003s) [upload=272455]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 18 14:11:00.893465627 (310h42m0.764548672s) [upload=272455]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 18 14:11:01.215063464 (310h42m1.086146509s) [upload=272455]: Done uploading segments... Dec 18 14:11:01.215089965 (310h42m1.08617301s) [upload=272455]: Upload finished. err: <nil> Duration: 5.785244282s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 17 out of 60 is failed (threshold=6) [....XXX.....X..X.X..............X.......X.....X.X.XX...XXXXX] duration: 5.001488273s trace: https://jaeger.dev.storj.io/trace/4e826141d6998096 FAIL (2025-12-18 14:25:04 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001514854s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)