Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 5.896150094s trace: https://jaeger.dev.storj.io/trace/1ad10e1389335c60 FAIL (2025-12-09 12:03:41 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.896177436s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 16.060919148s trace: https://jaeger.dev.storj.io/trace/180c8a02a447001f FAIL (2025-12-09 12:03:33 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 1m24.245559s) [upload=79956]: Starting upload Dec 9 11:10:24.374550547 (91h41m24.245633592s) [upload=79956]: Done starting upload Dec 9 11:10:24.374695774 (91h41m24.245778819s) [upload=79956]: Uploading segments... Dec 9 11:10:24.374799832 (91h41m24.245882877s) [upload=79956]: Got next segment. Inline: false Dec 9 11:10:24.374845269 (91h41m24.245928314s) [upload=79956]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 11:10:24.374891817 (91h41m24.245974862s) [upload=79956]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 11:10:24.692953929 (91h41m24.564036964s) [seg_pos={0 0}, upload=79956]: Begin upload segment... Dec 9 11:10:24.692996710 (91h41m24.564079745s) [seg_pos={0 0}, upload=79956]: Done begin upload segment. Dec 9 11:10:24.693021047 (91h41m24.564104092s) [noise=true , address=145.79.227.42:4121 , piece_id=THP22HNINRQMLRTQJKBHRCSBE5O4JMZWVNFRTJDVFCP7CKP57V7Q, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:24.693030324 (91h41m24.56411336s) [noise=true , address=145.79.227.28:4727 , piece_id=2VL43MLR2PE4BPHW4IWKNWAW3PSOAX2EV32XWLQYMYBM4FZ7RT6Q, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:24.693620949 (91h41m24.564703994s) [noise=true , address=145.79.227.23:4545 , piece_id=XVIQ424I6HQC4IXH2XAYRG7DFXJ7LJ44VT7ZCWVHFFUTXMTBZYXQ, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:24.694151741 (91h41m24.565234786s) [noise=true , address=145.79.227.41:4087 , piece_id=CC643HDKTLAEAAOHWOSXJL75BJH62EL7QISOZVG2VYPH4S3LDKMQ, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:24.694508199 (91h41m24.565591244s) [noise=true , address=145.79.227.39:4014 , piece_id=4F3Y2MMBAZVYJW6T5ZRNLQ4ZU56GF4EQMY6BPKZTCJ7KR7EY6UXQ, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:24.694727136 (91h41m24.565810181s) [noise=true , address=45.91.116.4:4939 , piece_id=QK4MQXBPU7SZ7SUA7SMDF5IJ2UUA6Q6MA2MA4UR3DSMU6QN7OHSQ, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:24.695257175 (91h41m24.56634022s) [noise=true , address=145.79.227.26:4655 , piece_id=WSNSSYE2G6LWCKNBMS63KSEPZBQSENSZ536OHOEAHKMLAIVQA4ZA, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:24.695748691 (91h41m24.566831736s) [noise=true , address=145.79.227.27:4689 , piece_id=4PFOTYEIDSKPMQF4LFVZZB5MK4O6JXFTU2QCLC5GUZTAUFPOE45Q, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:24.696362650 (91h41m24.567445695s) [noise=true , address=145.79.227.31:4799 , piece_id=4GDZXDMIOOWAIEKGEVXHGKF2LQ4YLSCCPA4S6UPCCAVHGRUT4DCA, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:24.698217953 (91h41m24.569300988s) [noise=true , address=77.90.31.14:4661 , piece_id=LJY3BTAERCAOCCIKW2THHEBAYKOEXDYQVGJTMWHYBACZJ4XJTBBA, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:24.701427525 (91h41m24.57251057s) [noise=true , address=77.90.31.12:4592 , piece_id=THMPWJ2JV2RVPMFHR4UPYJEC2FT7AS7QYVZQI427KGYKTYGI3ZMQ, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:24.705981576 (91h41m24.577064621s) [noise=true , address=145.79.227.33:4869 , piece_id=R7N7EKAQ6PUL7IBCY7SH4GTGQ6XYU3ZB7UJUVKFX5S7NTFRSCFNA, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:24.708340077 (91h41m24.579423112s) [noise=true , address=145.79.227.27:4690 , piece_id=OBIAVWCVVSLUDUW7G7ZGIQRIVOHO5R4KR25AVMHNA4RSTT6KB23Q, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:24.710358000 (91h41m24.581441035s) [noise=true , address=145.79.227.43:4157 , piece_id=7WZ2NUCKMURKNDQCZ6FHUC2I4XFXK4W3B3DR26DK6F2G62R7YC7Q, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:24.711000353 (91h41m24.582083388s) [noise=true , address=145.79.227.29:4943 , piece_id=PVDPIUVHEL3DLK45G2NTR3INZJ2CEPKLF6KX6CZMOMKXEOFRTTBA, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:24.712392574 (91h41m24.583475609s) [noise=true , address=77.90.31.16:4734 , piece_id=BQYJNHPRU7WKEHXUXJCUX6ZDU2RSVDPAAJTOD3VEQCEGE7TVR6CQ, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:26.323507311 (91h41m26.194590346s) [upload=79956]: Next returned nil segment Dec 9 11:10:26.323527019 (91h41m26.194610064s) [upload=79956]: Waiting for error group managing segments... Dec 9 11:10:26.705924029 (91h41m26.577007074s) [noise=true , address=145.79.227.28:4727 , piece_id=2VL43MLR2PE4BPHW4IWKNWAW3PSOAX2EV32XWLQYMYBM4FZ7RT6Q, seg_pos={0 0}, upload=79956]: Done uploading piece. err: <nil> Dec 9 11:10:26.713527548 (91h41m26.584610593s) [noise=true , address=145.79.227.42:4121 , piece_id=THP22HNINRQMLRTQJKBHRCSBE5O4JMZWVNFRTJDVFCP7CKP57V7Q, seg_pos={0 0}, upload=79956]: Done uploading piece. err: <nil> Dec 9 11:10:26.715957385 (91h41m26.58704043s) [noise=true , address=145.79.227.23:4545 , piece_id=XVIQ424I6HQC4IXH2XAYRG7DFXJ7LJ44VT7ZCWVHFFUTXMTBZYXQ, seg_pos={0 0}, upload=79956]: Done uploading piece. err: <nil> Dec 9 11:10:26.721154390 (91h41m26.592237435s) [noise=true , address=145.79.227.26:4655 , piece_id=WSNSSYE2G6LWCKNBMS63KSEPZBQSENSZ536OHOEAHKMLAIVQA4ZA, seg_pos={0 0}, upload=79956]: Done uploading piece. err: <nil> Dec 9 11:10:26.726209116 (91h41m26.597292161s) [noise=true , address=145.79.227.43:4157 , piece_id=7WZ2NUCKMURKNDQCZ6FHUC2I4XFXK4W3B3DR26DK6F2G62R7YC7Q, seg_pos={0 0}, upload=79956]: Done uploading piece. err: <nil> Dec 9 11:10:26.744717479 (91h41m26.615800524s) [noise=true , address=145.79.227.31:4799 , piece_id=4GDZXDMIOOWAIEKGEVXHGKF2LQ4YLSCCPA4S6UPCCAVHGRUT4DCA, seg_pos={0 0}, upload=79956]: Done uploading piece. err: <nil> Dec 9 11:10:26.865540120 (91h41m26.736623165s) [noise=true , address=77.90.31.14:4661 , piece_id=LJY3BTAERCAOCCIKW2THHEBAYKOEXDYQVGJTMWHYBACZJ4XJTBBA, seg_pos={0 0}, upload=79956]: Done uploading piece. err: <nil> Dec 9 11:10:26.874733556 (91h41m26.745816591s) [noise=true , address=77.90.31.12:4592 , piece_id=THMPWJ2JV2RVPMFHR4UPYJEC2FT7AS7QYVZQI427KGYKTYGI3ZMQ, seg_pos={0 0}, upload=79956]: Done uploading piece. err: <nil> Dec 9 11:10:27.552332367 (91h41m27.423415412s) [noise=true , address=145.79.227.39:4014 , piece_id=4F3Y2MMBAZVYJW6T5ZRNLQ4ZU56GF4EQMY6BPKZTCJ7KR7EY6UXQ, seg_pos={0 0}, upload=79956]: Done uploading piece. err: <nil> Dec 9 11:10:27.555135495 (91h41m27.42621853s) [noise=true , address=145.79.227.27:4689 , piece_id=4PFOTYEIDSKPMQF4LFVZZB5MK4O6JXFTU2QCLC5GUZTAUFPOE45Q, seg_pos={0 0}, upload=79956]: Done uploading piece. err: <nil> Dec 9 11:10:27.837239356 (91h41m27.708322401s) [noise=true , address=145.79.227.41:4087 , piece_id=CC643HDKTLAEAAOHWOSXJL75BJH62EL7QISOZVG2VYPH4S3LDKMQ, seg_pos={0 0}, upload=79956]: Done uploading piece. err: <nil> Dec 9 11:10:27.850249129 (91h41m27.721332174s) [noise=true , address=145.79.227.29:4943 , piece_id=PVDPIUVHEL3DLK45G2NTR3INZJ2CEPKLF6KX6CZMOMKXEOFRTTBA, seg_pos={0 0}, upload=79956]: Done uploading piece. err: <nil> Dec 9 11:10:27.850441505 (91h41m27.72152455s) [noise=true , address=145.79.227.33:4869 , piece_id=R7N7EKAQ6PUL7IBCY7SH4GTGQ6XYU3ZB7UJUVKFX5S7NTFRSCFNA, seg_pos={0 0}, upload=79956]: Done uploading piece. err: <nil> Dec 9 11:10:27.850450944 (91h41m27.721533989s) [seg_pos={0 0}, upload=79956]: Segment reached optimal threshold of 13 pieces. Dec 9 11:10:27.850729945 (91h41m27.72181298s) [noise=true , address=77.90.31.16:4734 , piece_id=BQYJNHPRU7WKEHXUXJCUX6ZDU2RSVDPAAJTOD3VEQCEGE7TVR6CQ, seg_pos={0 0}, upload=79956]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: write tcp 188.166.225.170:57470->77.90.31.16:4734: use of closed network connection; write tcp 188.166.225.170:57470->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57470->77.90.31.16:4734: use of closed network connection Dec 9 11:10:27.850737028 (91h41m27.721820063s) [noise=true , address=77.90.31.16:4734 , piece_id=BQYJNHPRU7WKEHXUXJCUX6ZDU2RSVDPAAJTOD3VEQCEGE7TVR6CQ, seg_pos={0 0}, upload=79956]: Long tail cancellation due to optimal threshold Dec 9 11:10:27.850945425 (91h41m27.72202848s) [noise=true , address=45.91.116.4:4939 , piece_id=QK4MQXBPU7SZ7SUA7SMDF5IJ2UUA6Q6MA2MA4UR3DSMU6QN7OHSQ, seg_pos={0 0}, upload=79956]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: read tcp 188.166.225.170:48538->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 9 11:10:27.850974751 (91h41m27.722057816s) [noise=true , address=45.91.116.4:4939 , piece_id=QK4MQXBPU7SZ7SUA7SMDF5IJ2UUA6Q6MA2MA4UR3DSMU6QN7OHSQ, seg_pos={0 0}, upload=79956]: Long tail cancellation due to optimal threshold Dec 9 11:10:27.851168249 (91h41m27.722251294s) [noise=true , address=145.79.227.27:4690 , piece_id=OBIAVWCVVSLUDUW7G7ZGIQRIVOHO5R4KR25AVMHNA4RSTT6KB23Q, seg_pos={0 0}, upload=79956]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H): protocol: expected piece hash; context canceled; EOF Dec 9 11:10:27.851178119 (91h41m27.722261194s) [noise=true , address=145.79.227.27:4690 , piece_id=OBIAVWCVVSLUDUW7G7ZGIQRIVOHO5R4KR25AVMHNA4RSTT6KB23Q, seg_pos={0 0}, upload=79956]: Long tail cancellation due to optimal threshold Dec 9 11:10:27.851202003 (91h41m27.722285048s) [seg_pos={0 0}, upload=79956]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 9 11:10:27.851316833 (91h41m27.722399908s) [upload=79956]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 11:10:27.851325489 (91h41m27.722408534s) [upload=79956]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 11:25:29.822344337 (91h56m29.693427372s) [upload=79956]: Done uploading segments... Dec 9 11:25:29.822415463 (91h56m29.693498498s) [upload=79956]: Upload finished. err: metaclient: context deadline exceeded Duration: 16.060957781s core uplink us-select-1
checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 472.816218ms remote: "tcp:35.207.121.91:443" target: https://eu1.storj.io/login trace: https://jaeger.dev.storj.io/trace/215c1ddb7fa82bd3 FAIL (2025-12-09 12:03:26 UTC) orbiter warning: Get "https://eu1.storj.io/login": EOF Duration: 472.841777ms core satellite eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 10.922757274s trace: https://jaeger.dev.storj.io/trace/180c8a02a447001f FAIL (2025-12-09 12:03:05 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 1m24.245559s) [upload=79956]: Starting upload Dec 9 11:10:24.374550547 (91h41m24.245633592s) [upload=79956]: Done starting upload Dec 9 11:10:24.374695774 (91h41m24.245778819s) [upload=79956]: Uploading segments... Dec 9 11:10:24.374799832 (91h41m24.245882877s) [upload=79956]: Got next segment. Inline: false Dec 9 11:10:24.374845269 (91h41m24.245928314s) [upload=79956]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 11:10:24.374891817 (91h41m24.245974862s) [upload=79956]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 11:10:24.692953929 (91h41m24.564036964s) [seg_pos={0 0}, upload=79956]: Begin upload segment... Dec 9 11:10:24.692996710 (91h41m24.564079745s) [seg_pos={0 0}, upload=79956]: Done begin upload segment. Dec 9 11:10:24.693021047 (91h41m24.564104092s) [noise=true , address=145.79.227.42:4121 , piece_id=THP22HNINRQMLRTQJKBHRCSBE5O4JMZWVNFRTJDVFCP7CKP57V7Q, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:24.693030324 (91h41m24.56411336s) [noise=true , address=145.79.227.28:4727 , piece_id=2VL43MLR2PE4BPHW4IWKNWAW3PSOAX2EV32XWLQYMYBM4FZ7RT6Q, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:24.693620949 (91h41m24.564703994s) [noise=true , address=145.79.227.23:4545 , piece_id=XVIQ424I6HQC4IXH2XAYRG7DFXJ7LJ44VT7ZCWVHFFUTXMTBZYXQ, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:24.694151741 (91h41m24.565234786s) [noise=true , address=145.79.227.41:4087 , piece_id=CC643HDKTLAEAAOHWOSXJL75BJH62EL7QISOZVG2VYPH4S3LDKMQ, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:24.694508199 (91h41m24.565591244s) [noise=true , address=145.79.227.39:4014 , piece_id=4F3Y2MMBAZVYJW6T5ZRNLQ4ZU56GF4EQMY6BPKZTCJ7KR7EY6UXQ, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:24.694727136 (91h41m24.565810181s) [noise=true , address=45.91.116.4:4939 , piece_id=QK4MQXBPU7SZ7SUA7SMDF5IJ2UUA6Q6MA2MA4UR3DSMU6QN7OHSQ, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:24.695257175 (91h41m24.56634022s) [noise=true , address=145.79.227.26:4655 , piece_id=WSNSSYE2G6LWCKNBMS63KSEPZBQSENSZ536OHOEAHKMLAIVQA4ZA, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:24.695748691 (91h41m24.566831736s) [noise=true , address=145.79.227.27:4689 , piece_id=4PFOTYEIDSKPMQF4LFVZZB5MK4O6JXFTU2QCLC5GUZTAUFPOE45Q, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:24.696362650 (91h41m24.567445695s) [noise=true , address=145.79.227.31:4799 , piece_id=4GDZXDMIOOWAIEKGEVXHGKF2LQ4YLSCCPA4S6UPCCAVHGRUT4DCA, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:24.698217953 (91h41m24.569300988s) [noise=true , address=77.90.31.14:4661 , piece_id=LJY3BTAERCAOCCIKW2THHEBAYKOEXDYQVGJTMWHYBACZJ4XJTBBA, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:24.701427525 (91h41m24.57251057s) [noise=true , address=77.90.31.12:4592 , piece_id=THMPWJ2JV2RVPMFHR4UPYJEC2FT7AS7QYVZQI427KGYKTYGI3ZMQ, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:24.705981576 (91h41m24.577064621s) [noise=true , address=145.79.227.33:4869 , piece_id=R7N7EKAQ6PUL7IBCY7SH4GTGQ6XYU3ZB7UJUVKFX5S7NTFRSCFNA, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:24.708340077 (91h41m24.579423112s) [noise=true , address=145.79.227.27:4690 , piece_id=OBIAVWCVVSLUDUW7G7ZGIQRIVOHO5R4KR25AVMHNA4RSTT6KB23Q, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:24.710358000 (91h41m24.581441035s) [noise=true , address=145.79.227.43:4157 , piece_id=7WZ2NUCKMURKNDQCZ6FHUC2I4XFXK4W3B3DR26DK6F2G62R7YC7Q, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:24.711000353 (91h41m24.582083388s) [noise=true , address=145.79.227.29:4943 , piece_id=PVDPIUVHEL3DLK45G2NTR3INZJ2CEPKLF6KX6CZMOMKXEOFRTTBA, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:24.712392574 (91h41m24.583475609s) [noise=true , address=77.90.31.16:4734 , piece_id=BQYJNHPRU7WKEHXUXJCUX6ZDU2RSVDPAAJTOD3VEQCEGE7TVR6CQ, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:26.323507311 (91h41m26.194590346s) [upload=79956]: Next returned nil segment Dec 9 11:10:26.323527019 (91h41m26.194610064s) [upload=79956]: Waiting for error group managing segments... Dec 9 11:10:26.705924029 (91h41m26.577007074s) [noise=true , address=145.79.227.28:4727 , piece_id=2VL43MLR2PE4BPHW4IWKNWAW3PSOAX2EV32XWLQYMYBM4FZ7RT6Q, seg_pos={0 0}, upload=79956]: Done uploading piece. err: <nil> Dec 9 11:10:26.713527548 (91h41m26.584610593s) [noise=true , address=145.79.227.42:4121 , piece_id=THP22HNINRQMLRTQJKBHRCSBE5O4JMZWVNFRTJDVFCP7CKP57V7Q, seg_pos={0 0}, upload=79956]: Done uploading piece. err: <nil> Dec 9 11:10:26.715957385 (91h41m26.58704043s) [noise=true , address=145.79.227.23:4545 , piece_id=XVIQ424I6HQC4IXH2XAYRG7DFXJ7LJ44VT7ZCWVHFFUTXMTBZYXQ, seg_pos={0 0}, upload=79956]: Done uploading piece. err: <nil> Dec 9 11:10:26.721154390 (91h41m26.592237435s) [noise=true , address=145.79.227.26:4655 , piece_id=WSNSSYE2G6LWCKNBMS63KSEPZBQSENSZ536OHOEAHKMLAIVQA4ZA, seg_pos={0 0}, upload=79956]: Done uploading piece. err: <nil> Dec 9 11:10:26.726209116 (91h41m26.597292161s) [noise=true , address=145.79.227.43:4157 , piece_id=7WZ2NUCKMURKNDQCZ6FHUC2I4XFXK4W3B3DR26DK6F2G62R7YC7Q, seg_pos={0 0}, upload=79956]: Done uploading piece. err: <nil> Dec 9 11:10:26.744717479 (91h41m26.615800524s) [noise=true , address=145.79.227.31:4799 , piece_id=4GDZXDMIOOWAIEKGEVXHGKF2LQ4YLSCCPA4S6UPCCAVHGRUT4DCA, seg_pos={0 0}, upload=79956]: Done uploading piece. err: <nil> Dec 9 11:10:26.865540120 (91h41m26.736623165s) [noise=true , address=77.90.31.14:4661 , piece_id=LJY3BTAERCAOCCIKW2THHEBAYKOEXDYQVGJTMWHYBACZJ4XJTBBA, seg_pos={0 0}, upload=79956]: Done uploading piece. err: <nil> Dec 9 11:10:26.874733556 (91h41m26.745816591s) [noise=true , address=77.90.31.12:4592 , piece_id=THMPWJ2JV2RVPMFHR4UPYJEC2FT7AS7QYVZQI427KGYKTYGI3ZMQ, seg_pos={0 0}, upload=79956]: Done uploading piece. err: <nil> Dec 9 11:10:27.552332367 (91h41m27.423415412s) [noise=true , address=145.79.227.39:4014 , piece_id=4F3Y2MMBAZVYJW6T5ZRNLQ4ZU56GF4EQMY6BPKZTCJ7KR7EY6UXQ, seg_pos={0 0}, upload=79956]: Done uploading piece. err: <nil> Dec 9 11:10:27.555135495 (91h41m27.42621853s) [noise=true , address=145.79.227.27:4689 , piece_id=4PFOTYEIDSKPMQF4LFVZZB5MK4O6JXFTU2QCLC5GUZTAUFPOE45Q, seg_pos={0 0}, upload=79956]: Done uploading piece. err: <nil> Dec 9 11:10:27.837239356 (91h41m27.708322401s) [noise=true , address=145.79.227.41:4087 , piece_id=CC643HDKTLAEAAOHWOSXJL75BJH62EL7QISOZVG2VYPH4S3LDKMQ, seg_pos={0 0}, upload=79956]: Done uploading piece. err: <nil> Dec 9 11:10:27.850249129 (91h41m27.721332174s) [noise=true , address=145.79.227.29:4943 , piece_id=PVDPIUVHEL3DLK45G2NTR3INZJ2CEPKLF6KX6CZMOMKXEOFRTTBA, seg_pos={0 0}, upload=79956]: Done uploading piece. err: <nil> Dec 9 11:10:27.850441505 (91h41m27.72152455s) [noise=true , address=145.79.227.33:4869 , piece_id=R7N7EKAQ6PUL7IBCY7SH4GTGQ6XYU3ZB7UJUVKFX5S7NTFRSCFNA, seg_pos={0 0}, upload=79956]: Done uploading piece. err: <nil> Dec 9 11:10:27.850450944 (91h41m27.721533989s) [seg_pos={0 0}, upload=79956]: Segment reached optimal threshold of 13 pieces. Dec 9 11:10:27.850729945 (91h41m27.72181298s) [noise=true , address=77.90.31.16:4734 , piece_id=BQYJNHPRU7WKEHXUXJCUX6ZDU2RSVDPAAJTOD3VEQCEGE7TVR6CQ, seg_pos={0 0}, upload=79956]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: write tcp 188.166.225.170:57470->77.90.31.16:4734: use of closed network connection; write tcp 188.166.225.170:57470->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57470->77.90.31.16:4734: use of closed network connection Dec 9 11:10:27.850737028 (91h41m27.721820063s) [noise=true , address=77.90.31.16:4734 , piece_id=BQYJNHPRU7WKEHXUXJCUX6ZDU2RSVDPAAJTOD3VEQCEGE7TVR6CQ, seg_pos={0 0}, upload=79956]: Long tail cancellation due to optimal threshold Dec 9 11:10:27.850945425 (91h41m27.72202848s) [noise=true , address=45.91.116.4:4939 , piece_id=QK4MQXBPU7SZ7SUA7SMDF5IJ2UUA6Q6MA2MA4UR3DSMU6QN7OHSQ, seg_pos={0 0}, upload=79956]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: read tcp 188.166.225.170:48538->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 9 11:10:27.850974751 (91h41m27.722057816s) [noise=true , address=45.91.116.4:4939 , piece_id=QK4MQXBPU7SZ7SUA7SMDF5IJ2UUA6Q6MA2MA4UR3DSMU6QN7OHSQ, seg_pos={0 0}, upload=79956]: Long tail cancellation due to optimal threshold Dec 9 11:10:27.851168249 (91h41m27.722251294s) [noise=true , address=145.79.227.27:4690 , piece_id=OBIAVWCVVSLUDUW7G7ZGIQRIVOHO5R4KR25AVMHNA4RSTT6KB23Q, seg_pos={0 0}, upload=79956]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H): protocol: expected piece hash; context canceled; EOF Dec 9 11:10:27.851178119 (91h41m27.722261194s) [noise=true , address=145.79.227.27:4690 , piece_id=OBIAVWCVVSLUDUW7G7ZGIQRIVOHO5R4KR25AVMHNA4RSTT6KB23Q, seg_pos={0 0}, upload=79956]: Long tail cancellation due to optimal threshold Dec 9 11:10:27.851202003 (91h41m27.722285048s) [seg_pos={0 0}, upload=79956]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 9 11:10:27.851316833 (91h41m27.722399908s) [upload=79956]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 11:10:27.851325489 (91h41m27.722408534s) [upload=79956]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 11:25:29.822344337 (91h56m29.693427372s) [upload=79956]: Done uploading segments... Dec 9 11:25:29.822415463 (91h56m29.693498498s) [upload=79956]: Upload finished. err: metaclient: context deadline exceeded Duration: 10.922785467s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [X............X....................XX................X.....X.] duration: 3.464500841s trace: https://jaeger.dev.storj.io/trace/9c5f1cb27c256e3 FAIL (2025-12-09 12:02:54 UTC) orbiter warning: context deadline exceeded Duration: 3.464524858s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 14.26473981s trace: https://jaeger.dev.storj.io/trace/7098c007ca020436 FAIL (2025-12-09 12:02:46 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , address=185.167.204.205:4029 , piece_id=WMBEVUF62ZU76TQDICPDPMG4JA7374YQFXTI5XJFSN6MJZBATS7Q, seg_pos={0 0}, upload=80368]: Uploading piece... Dec 9 11:38:49.362641042 (92h9m49.233724077s) [noise=true , address=185.167.204.206:4037 , piece_id=LO332KWEY4NLBKVGEVME2SF7UIGTEVCKLOHRCC7FLIJEPP5DTKTA, seg_pos={0 0}, upload=80368]: Uploading piece... Dec 9 11:38:49.362747554 (92h9m49.233830599s) [noise=true , address=185.167.204.206:4045 , piece_id=7RI3JKTQDSVERWTVQ4NIGZMBNAC6OPYUCTPU77HX3V74NBG7XALQ, seg_pos={0 0}, upload=80368]: Uploading piece... Dec 9 11:38:49.363050582 (92h9m49.234133627s) [noise=true , address=185.167.204.206:4059 , piece_id=HLZEIABVICZTZCKXHBKIFQJ3UVRGVV5EWHWFFL5G3E7CYGJN4GHA, seg_pos={0 0}, upload=80368]: Uploading piece... Dec 9 11:38:49.363113251 (92h9m49.234196286s) [noise=true , address=185.167.204.205:4016 , piece_id=55J2Z5J7LPDPCBKXAQ7V4BQW4U7W6UAQVF63LYH3BPFHSEQE6CEA, seg_pos={0 0}, upload=80368]: Uploading piece... Dec 9 11:38:49.363383676 (92h9m49.234466711s) [noise=true , address=185.167.204.206:4048 , piece_id=XMV7K5EEISKMT5JQFUD6EXZ3EYELHBWF6Y3J276VUUP2YGXJKF3A, seg_pos={0 0}, upload=80368]: Uploading piece... Dec 9 11:38:49.363681573 (92h9m49.234764608s) [noise=true , address=185.167.204.205:4001 , piece_id=KIUIPM26OHXISSEHSZPLFHUXF4NVOHSY6TILREFVGDRDBB5IOCQQ, seg_pos={0 0}, upload=80368]: Uploading piece... Dec 9 11:38:49.364037821 (92h9m49.235120866s) [noise=true , address=81.171.14.70:4042 , piece_id=WWJDEHBRMRVXANEWVG6B6QLD2SR7M6FXH4642UZ3AFQSDGCO2HYA, seg_pos={0 0}, upload=80368]: Uploading piece... Dec 9 11:38:49.364321191 (92h9m49.235404226s) [noise=true , address=81.171.14.70:4032 , piece_id=GOAKND7Z3IDBUYT3ILUUBU7YC3UMA6PPDJLLMFTH2OKHF76GCHWA, seg_pos={0 0}, upload=80368]: Uploading piece... Dec 9 11:38:49.364596545 (92h9m49.23567959s) [noise=true , address=81.171.14.70:4039 , piece_id=EJYQEP7LA6QJQICJQQVTNIXDN6ID5PAPL54GADSICUBJ3YQQ5E7Q, seg_pos={0 0}, upload=80368]: Uploading piece... Dec 9 11:38:49.365194575 (92h9m49.23627762s) [noise=true , address=89.149.201.147:4011 , piece_id=IJZKML6ZS5AF7KRYSHG4ZDR7AZSJGUWRSPTLSYTWR6WKYXWORAYA, seg_pos={0 0}, upload=80368]: Uploading piece... Dec 9 11:38:50.188365108 (92h9m50.059448153s) [upload=80368]: Next returned nil segment Dec 9 11:38:50.188379346 (92h9m50.059462391s) [upload=80368]: Waiting for error group managing segments... Dec 9 11:38:50.573198910 (92h9m50.444281955s) [noise=true , address=185.167.204.206:4062 , piece_id=UZUYTLORRUJOMEX62Z7ABCOXWZHBZ7PV72MA4TZIWDDWZKBR22OA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:50.577545046 (92h9m50.448628081s) [noise=true , address=185.167.204.206:4059 , piece_id=HLZEIABVICZTZCKXHBKIFQJ3UVRGVV5EWHWFFL5G3E7CYGJN4GHA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:50.577840359 (92h9m50.448923394s) [noise=true , address=185.167.204.206:4045 , piece_id=7RI3JKTQDSVERWTVQ4NIGZMBNAC6OPYUCTPU77HX3V74NBG7XALQ, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:50.578430443 (92h9m50.449513478s) [noise=true , address=185.167.204.206:4048 , piece_id=XMV7K5EEISKMT5JQFUD6EXZ3EYELHBWF6Y3J276VUUP2YGXJKF3A, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:50.579912414 (92h9m50.450995459s) [noise=true , address=185.167.204.205:4016 , piece_id=55J2Z5J7LPDPCBKXAQ7V4BQW4U7W6UAQVF63LYH3BPFHSEQE6CEA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:50.580088579 (92h9m50.451171625s) [noise=true , address=185.167.204.206:4068 , piece_id=FDF7EQNPFGNEDO6CEPYGMAAJOIT4TPHY2PVYHNXH7ILC7AY2LO3A, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:50.582475435 (92h9m50.45355848s) [noise=true , address=185.167.204.206:4037 , piece_id=LO332KWEY4NLBKVGEVME2SF7UIGTEVCKLOHRCC7FLIJEPP5DTKTA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:50.586259020 (92h9m50.457342055s) [noise=true , address=185.167.204.205:4030 , piece_id=VHBCZ6QQRBZ4THX6XCFIZ6JSFMIONQFSEZ3ERGOZB6L7L5USAKMA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:50.586458941 (92h9m50.457541986s) [noise=true , address=185.167.204.206:4060 , piece_id=A3NTK6DGBBOACMUONBGII27GVS7QZWOGDEKBPZFX76RSH5HMXS5A, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:50.588436907 (92h9m50.459519942s) [noise=true , address=185.167.204.205:4009 , piece_id=5WUDBEGUENMGEMSND2VCENBNGG2BIV47ICSWN3CJYLQQ3XIVZXKQ, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:50.589577038 (92h9m50.460660083s) [noise=true , address=185.167.204.205:4029 , piece_id=WMBEVUF62ZU76TQDICPDPMG4JA7374YQFXTI5XJFSN6MJZBATS7Q, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:50.592840462 (92h9m50.463923507s) [noise=true , address=185.167.204.205:4001 , piece_id=KIUIPM26OHXISSEHSZPLFHUXF4NVOHSY6TILREFVGDRDBB5IOCQQ, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.141501915 (92h9m51.01258496s) [noise=true , address=81.171.14.70:4039 , piece_id=EJYQEP7LA6QJQICJQQVTNIXDN6ID5PAPL54GADSICUBJ3YQQ5E7Q, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.153777871 (92h9m51.024860916s) [noise=true , address=81.171.14.70:4042 , piece_id=WWJDEHBRMRVXANEWVG6B6QLD2SR7M6FXH4642UZ3AFQSDGCO2HYA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.153948696 (92h9m51.025031751s) [noise=true , address=81.171.14.70:4028 , piece_id=NSYGSKJBIHMHZZX7HK7N3UQDGKWWQARJPGLAWH2TEPGWKNYWPC4A, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.157888027 (92h9m51.028971062s) [noise=true , address=81.171.14.70:4041 , piece_id=PZNQJLGVYH3CBPX4L2URCEVLMGK4JLQAMVFX25UBONKVYSTSEEWQ, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.170799434 (92h9m51.041882489s) [noise=true , address=81.171.14.70:4045 , piece_id=TZIZNHRR66JMWMZJNUHYFPYJ2ORR3OLKZ7VOQ5MIWXQOFARHL4VA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.171007360 (92h9m51.042090405s) [noise=true , address=81.171.14.70:4035 , piece_id=5T5MX4YDMI4IUY4LDSREN2MTN3NUL4GS6KZXDMNZMKJYY3VMS47Q, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.184999573 (92h9m51.056082648s) [noise=true , address=81.171.14.70:4032 , piece_id=GOAKND7Z3IDBUYT3ILUUBU7YC3UMA6PPDJLLMFTH2OKHF76GCHWA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.196460517 (92h9m51.067543562s) [noise=true , address=81.171.14.70:4040 , piece_id=4GA5CWKM5RJBL2IMWZNRGAA2T2JTFX54AFEFI3NAV2GNTKIDUELQ, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.199311115 (92h9m51.07039416s) [noise=true , address=81.171.14.70:4037 , piece_id=G6MBAEOMB7R4MQKZDULZY4DYQ5NR2QMCKW6HWYA55VD5YMSAERPA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.237519992 (92h9m51.108603028s) [noise=true , address=81.171.14.70:4030 , piece_id=VFAGZYZMOZKEHRNPUR2D2JCRDZFUFXUOS3JTTI2UOUYTIUY74NFQ, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.733107833 (92h9m51.604190868s) [noise=true , address=89.149.201.147:4005 , piece_id=OCHDE5VQIQPUVCOFXX343C7LXUIJFD4Q62ZZPEXKRN63KARUQR5A, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.815947736 (92h9m51.687030771s) [noise=true , address=89.149.201.147:4015 , piece_id=PGXJFJBHXVMXBTIWDTAOZKAU2JKTFSGCAD5I5MWU2MD32GF3AMDQ, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.841035780 (92h9m51.712118825s) [noise=true , address=89.149.201.147:4020 , piece_id=KIYFOE4WT3CGQQ3CL3BP2A3HH2KVKKLAUUGMHYPADVKI7BVBBRTA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.864720530 (92h9m51.735803575s) [noise=true , address=89.149.201.147:4002 , piece_id=TIK5YZA7N6MYGZ6KGNXPFBEQXD3DE2UXQ5Q2JP3EZGSLJ7TAO2GA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.864741660 (92h9m51.735824705s) [seg_pos={0 0}, upload=80368]: Segment reached optimal threshold of 26 pieces. Dec 9 11:38:51.865018136 (92h9m51.736101181s) [noise=true , address=89.149.201.147:4011 , piece_id=IJZKML6ZS5AF7KRYSHG4ZDR7AZSJGUWRSPTLSYTWR6WKYXWORAYA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PcxodnQMJERGjvBgnz3iEiC9MseRhMydkCriwGpQKqi6DGFDg): protocol: expected piece hash; context canceled; EOF Dec 9 11:38:51.865029489 (92h9m51.736112534s) [noise=true , address=89.149.201.147:4011 , piece_id=IJZKML6ZS5AF7KRYSHG4ZDR7AZSJGUWRSPTLSYTWR6WKYXWORAYA, seg_pos={0 0}, upload=80368]: Long tail cancellation due to optimal threshold Dec 9 11:38:51.865308640 (92h9m51.736391685s) [noise=true , address=89.149.201.147:4007 , piece_id=FHFZMTRZEYLGK2QOXICD5VLT77MIFP3HL6Z7SP7URQEY5L3HIDTA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Q7d4vj1KoWhDr6xd9gEfFrZ1sV3Yro2AFke2f3Ke1rXsGnvew): protocol: expected piece hash; context canceled; EOF Dec 9 11:38:51.865331703 (92h9m51.736414748s) [noise=true , address=89.149.201.147:4007 , piece_id=FHFZMTRZEYLGK2QOXICD5VLT77MIFP3HL6Z7SP7URQEY5L3HIDTA, seg_pos={0 0}, upload=80368]: Long tail cancellation due to optimal threshold Dec 9 11:38:51.865403281 (92h9m51.736486326s) [seg_pos={0 0}, upload=80368]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 9 11:38:51.865634440 (92h9m51.736717485s) [upload=80368]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 11:38:51.865643307 (92h9m51.736726352s) [upload=80368]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 11:40:37.870707645 (92h11m37.741790681s) [upload=80368]: Done uploading segments... Dec 9 11:40:37.870754294 (92h11m37.741837339s) [upload=80368]: Upload finished. err: metaclient: context deadline exceeded Duration: 14.264774576s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 5.890637062s trace: https://jaeger.dev.storj.io/trace/1ad10e1389335c60 FAIL (2025-12-09 12:02:42 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.890662021s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 466.77781ms remote: "tcp:35.207.121.91:443" target: https://eu1.storj.io/login trace: https://jaeger.dev.storj.io/trace/215c1ddb7fa82bd3 FAIL (2025-12-09 12:02:41 UTC) orbiter warning: Get "https://eu1.storj.io/login": EOF Duration: 466.812285ms core satellite eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 7.379845826s trace: https://jaeger.dev.storj.io/trace/180c8a02a447001f FAIL (2025-12-09 12:02:22 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 1m24.245559s) [upload=79956]: Starting upload Dec 9 11:10:24.374550547 (91h41m24.245633592s) [upload=79956]: Done starting upload Dec 9 11:10:24.374695774 (91h41m24.245778819s) [upload=79956]: Uploading segments... Dec 9 11:10:24.374799832 (91h41m24.245882877s) [upload=79956]: Got next segment. Inline: false Dec 9 11:10:24.374845269 (91h41m24.245928314s) [upload=79956]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 11:10:24.374891817 (91h41m24.245974862s) [upload=79956]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 11:10:24.692953929 (91h41m24.564036964s) [seg_pos={0 0}, upload=79956]: Begin upload segment... Dec 9 11:10:24.692996710 (91h41m24.564079745s) [seg_pos={0 0}, upload=79956]: Done begin upload segment. Dec 9 11:10:24.693021047 (91h41m24.564104092s) [noise=true , address=145.79.227.42:4121 , piece_id=THP22HNINRQMLRTQJKBHRCSBE5O4JMZWVNFRTJDVFCP7CKP57V7Q, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:24.693030324 (91h41m24.56411336s) [noise=true , address=145.79.227.28:4727 , piece_id=2VL43MLR2PE4BPHW4IWKNWAW3PSOAX2EV32XWLQYMYBM4FZ7RT6Q, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:24.693620949 (91h41m24.564703994s) [noise=true , address=145.79.227.23:4545 , piece_id=XVIQ424I6HQC4IXH2XAYRG7DFXJ7LJ44VT7ZCWVHFFUTXMTBZYXQ, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:24.694151741 (91h41m24.565234786s) [noise=true , address=145.79.227.41:4087 , piece_id=CC643HDKTLAEAAOHWOSXJL75BJH62EL7QISOZVG2VYPH4S3LDKMQ, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:24.694508199 (91h41m24.565591244s) [noise=true , address=145.79.227.39:4014 , piece_id=4F3Y2MMBAZVYJW6T5ZRNLQ4ZU56GF4EQMY6BPKZTCJ7KR7EY6UXQ, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:24.694727136 (91h41m24.565810181s) [noise=true , address=45.91.116.4:4939 , piece_id=QK4MQXBPU7SZ7SUA7SMDF5IJ2UUA6Q6MA2MA4UR3DSMU6QN7OHSQ, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:24.695257175 (91h41m24.56634022s) [noise=true , address=145.79.227.26:4655 , piece_id=WSNSSYE2G6LWCKNBMS63KSEPZBQSENSZ536OHOEAHKMLAIVQA4ZA, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:24.695748691 (91h41m24.566831736s) [noise=true , address=145.79.227.27:4689 , piece_id=4PFOTYEIDSKPMQF4LFVZZB5MK4O6JXFTU2QCLC5GUZTAUFPOE45Q, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:24.696362650 (91h41m24.567445695s) [noise=true , address=145.79.227.31:4799 , piece_id=4GDZXDMIOOWAIEKGEVXHGKF2LQ4YLSCCPA4S6UPCCAVHGRUT4DCA, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:24.698217953 (91h41m24.569300988s) [noise=true , address=77.90.31.14:4661 , piece_id=LJY3BTAERCAOCCIKW2THHEBAYKOEXDYQVGJTMWHYBACZJ4XJTBBA, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:24.701427525 (91h41m24.57251057s) [noise=true , address=77.90.31.12:4592 , piece_id=THMPWJ2JV2RVPMFHR4UPYJEC2FT7AS7QYVZQI427KGYKTYGI3ZMQ, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:24.705981576 (91h41m24.577064621s) [noise=true , address=145.79.227.33:4869 , piece_id=R7N7EKAQ6PUL7IBCY7SH4GTGQ6XYU3ZB7UJUVKFX5S7NTFRSCFNA, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:24.708340077 (91h41m24.579423112s) [noise=true , address=145.79.227.27:4690 , piece_id=OBIAVWCVVSLUDUW7G7ZGIQRIVOHO5R4KR25AVMHNA4RSTT6KB23Q, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:24.710358000 (91h41m24.581441035s) [noise=true , address=145.79.227.43:4157 , piece_id=7WZ2NUCKMURKNDQCZ6FHUC2I4XFXK4W3B3DR26DK6F2G62R7YC7Q, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:24.711000353 (91h41m24.582083388s) [noise=true , address=145.79.227.29:4943 , piece_id=PVDPIUVHEL3DLK45G2NTR3INZJ2CEPKLF6KX6CZMOMKXEOFRTTBA, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:24.712392574 (91h41m24.583475609s) [noise=true , address=77.90.31.16:4734 , piece_id=BQYJNHPRU7WKEHXUXJCUX6ZDU2RSVDPAAJTOD3VEQCEGE7TVR6CQ, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:26.323507311 (91h41m26.194590346s) [upload=79956]: Next returned nil segment Dec 9 11:10:26.323527019 (91h41m26.194610064s) [upload=79956]: Waiting for error group managing segments... Dec 9 11:10:26.705924029 (91h41m26.577007074s) [noise=true , address=145.79.227.28:4727 , piece_id=2VL43MLR2PE4BPHW4IWKNWAW3PSOAX2EV32XWLQYMYBM4FZ7RT6Q, seg_pos={0 0}, upload=79956]: Done uploading piece. err: <nil> Dec 9 11:10:26.713527548 (91h41m26.584610593s) [noise=true , address=145.79.227.42:4121 , piece_id=THP22HNINRQMLRTQJKBHRCSBE5O4JMZWVNFRTJDVFCP7CKP57V7Q, seg_pos={0 0}, upload=79956]: Done uploading piece. err: <nil> Dec 9 11:10:26.715957385 (91h41m26.58704043s) [noise=true , address=145.79.227.23:4545 , piece_id=XVIQ424I6HQC4IXH2XAYRG7DFXJ7LJ44VT7ZCWVHFFUTXMTBZYXQ, seg_pos={0 0}, upload=79956]: Done uploading piece. err: <nil> Dec 9 11:10:26.721154390 (91h41m26.592237435s) [noise=true , address=145.79.227.26:4655 , piece_id=WSNSSYE2G6LWCKNBMS63KSEPZBQSENSZ536OHOEAHKMLAIVQA4ZA, seg_pos={0 0}, upload=79956]: Done uploading piece. err: <nil> Dec 9 11:10:26.726209116 (91h41m26.597292161s) [noise=true , address=145.79.227.43:4157 , piece_id=7WZ2NUCKMURKNDQCZ6FHUC2I4XFXK4W3B3DR26DK6F2G62R7YC7Q, seg_pos={0 0}, upload=79956]: Done uploading piece. err: <nil> Dec 9 11:10:26.744717479 (91h41m26.615800524s) [noise=true , address=145.79.227.31:4799 , piece_id=4GDZXDMIOOWAIEKGEVXHGKF2LQ4YLSCCPA4S6UPCCAVHGRUT4DCA, seg_pos={0 0}, upload=79956]: Done uploading piece. err: <nil> Dec 9 11:10:26.865540120 (91h41m26.736623165s) [noise=true , address=77.90.31.14:4661 , piece_id=LJY3BTAERCAOCCIKW2THHEBAYKOEXDYQVGJTMWHYBACZJ4XJTBBA, seg_pos={0 0}, upload=79956]: Done uploading piece. err: <nil> Dec 9 11:10:26.874733556 (91h41m26.745816591s) [noise=true , address=77.90.31.12:4592 , piece_id=THMPWJ2JV2RVPMFHR4UPYJEC2FT7AS7QYVZQI427KGYKTYGI3ZMQ, seg_pos={0 0}, upload=79956]: Done uploading piece. err: <nil> Dec 9 11:10:27.552332367 (91h41m27.423415412s) [noise=true , address=145.79.227.39:4014 , piece_id=4F3Y2MMBAZVYJW6T5ZRNLQ4ZU56GF4EQMY6BPKZTCJ7KR7EY6UXQ, seg_pos={0 0}, upload=79956]: Done uploading piece. err: <nil> Dec 9 11:10:27.555135495 (91h41m27.42621853s) [noise=true , address=145.79.227.27:4689 , piece_id=4PFOTYEIDSKPMQF4LFVZZB5MK4O6JXFTU2QCLC5GUZTAUFPOE45Q, seg_pos={0 0}, upload=79956]: Done uploading piece. err: <nil> Dec 9 11:10:27.837239356 (91h41m27.708322401s) [noise=true , address=145.79.227.41:4087 , piece_id=CC643HDKTLAEAAOHWOSXJL75BJH62EL7QISOZVG2VYPH4S3LDKMQ, seg_pos={0 0}, upload=79956]: Done uploading piece. err: <nil> Dec 9 11:10:27.850249129 (91h41m27.721332174s) [noise=true , address=145.79.227.29:4943 , piece_id=PVDPIUVHEL3DLK45G2NTR3INZJ2CEPKLF6KX6CZMOMKXEOFRTTBA, seg_pos={0 0}, upload=79956]: Done uploading piece. err: <nil> Dec 9 11:10:27.850441505 (91h41m27.72152455s) [noise=true , address=145.79.227.33:4869 , piece_id=R7N7EKAQ6PUL7IBCY7SH4GTGQ6XYU3ZB7UJUVKFX5S7NTFRSCFNA, seg_pos={0 0}, upload=79956]: Done uploading piece. err: <nil> Dec 9 11:10:27.850450944 (91h41m27.721533989s) [seg_pos={0 0}, upload=79956]: Segment reached optimal threshold of 13 pieces. Dec 9 11:10:27.850729945 (91h41m27.72181298s) [noise=true , address=77.90.31.16:4734 , piece_id=BQYJNHPRU7WKEHXUXJCUX6ZDU2RSVDPAAJTOD3VEQCEGE7TVR6CQ, seg_pos={0 0}, upload=79956]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: write tcp 188.166.225.170:57470->77.90.31.16:4734: use of closed network connection; write tcp 188.166.225.170:57470->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57470->77.90.31.16:4734: use of closed network connection Dec 9 11:10:27.850737028 (91h41m27.721820063s) [noise=true , address=77.90.31.16:4734 , piece_id=BQYJNHPRU7WKEHXUXJCUX6ZDU2RSVDPAAJTOD3VEQCEGE7TVR6CQ, seg_pos={0 0}, upload=79956]: Long tail cancellation due to optimal threshold Dec 9 11:10:27.850945425 (91h41m27.72202848s) [noise=true , address=45.91.116.4:4939 , piece_id=QK4MQXBPU7SZ7SUA7SMDF5IJ2UUA6Q6MA2MA4UR3DSMU6QN7OHSQ, seg_pos={0 0}, upload=79956]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: read tcp 188.166.225.170:48538->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 9 11:10:27.850974751 (91h41m27.722057816s) [noise=true , address=45.91.116.4:4939 , piece_id=QK4MQXBPU7SZ7SUA7SMDF5IJ2UUA6Q6MA2MA4UR3DSMU6QN7OHSQ, seg_pos={0 0}, upload=79956]: Long tail cancellation due to optimal threshold Dec 9 11:10:27.851168249 (91h41m27.722251294s) [noise=true , address=145.79.227.27:4690 , piece_id=OBIAVWCVVSLUDUW7G7ZGIQRIVOHO5R4KR25AVMHNA4RSTT6KB23Q, seg_pos={0 0}, upload=79956]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H): protocol: expected piece hash; context canceled; EOF Dec 9 11:10:27.851178119 (91h41m27.722261194s) [noise=true , address=145.79.227.27:4690 , piece_id=OBIAVWCVVSLUDUW7G7ZGIQRIVOHO5R4KR25AVMHNA4RSTT6KB23Q, seg_pos={0 0}, upload=79956]: Long tail cancellation due to optimal threshold Dec 9 11:10:27.851202003 (91h41m27.722285048s) [seg_pos={0 0}, upload=79956]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 9 11:10:27.851316833 (91h41m27.722399908s) [upload=79956]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 11:10:27.851325489 (91h41m27.722408534s) [upload=79956]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 11:25:29.822344337 (91h56m29.693427372s) [upload=79956]: Done uploading segments... Dec 9 11:25:29.822415463 (91h56m29.693498498s) [upload=79956]: Upload finished. err: metaclient: context deadline exceeded Duration: 7.379867426s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 18.169864779s trace: https://jaeger.dev.storj.io/trace/7098c007ca020436 FAIL (2025-12-09 12:02:07 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , address=185.167.204.205:4029 , piece_id=WMBEVUF62ZU76TQDICPDPMG4JA7374YQFXTI5XJFSN6MJZBATS7Q, seg_pos={0 0}, upload=80368]: Uploading piece... Dec 9 11:38:49.362641042 (92h9m49.233724077s) [noise=true , address=185.167.204.206:4037 , piece_id=LO332KWEY4NLBKVGEVME2SF7UIGTEVCKLOHRCC7FLIJEPP5DTKTA, seg_pos={0 0}, upload=80368]: Uploading piece... Dec 9 11:38:49.362747554 (92h9m49.233830599s) [noise=true , address=185.167.204.206:4045 , piece_id=7RI3JKTQDSVERWTVQ4NIGZMBNAC6OPYUCTPU77HX3V74NBG7XALQ, seg_pos={0 0}, upload=80368]: Uploading piece... Dec 9 11:38:49.363050582 (92h9m49.234133627s) [noise=true , address=185.167.204.206:4059 , piece_id=HLZEIABVICZTZCKXHBKIFQJ3UVRGVV5EWHWFFL5G3E7CYGJN4GHA, seg_pos={0 0}, upload=80368]: Uploading piece... Dec 9 11:38:49.363113251 (92h9m49.234196286s) [noise=true , address=185.167.204.205:4016 , piece_id=55J2Z5J7LPDPCBKXAQ7V4BQW4U7W6UAQVF63LYH3BPFHSEQE6CEA, seg_pos={0 0}, upload=80368]: Uploading piece... Dec 9 11:38:49.363383676 (92h9m49.234466711s) [noise=true , address=185.167.204.206:4048 , piece_id=XMV7K5EEISKMT5JQFUD6EXZ3EYELHBWF6Y3J276VUUP2YGXJKF3A, seg_pos={0 0}, upload=80368]: Uploading piece... Dec 9 11:38:49.363681573 (92h9m49.234764608s) [noise=true , address=185.167.204.205:4001 , piece_id=KIUIPM26OHXISSEHSZPLFHUXF4NVOHSY6TILREFVGDRDBB5IOCQQ, seg_pos={0 0}, upload=80368]: Uploading piece... Dec 9 11:38:49.364037821 (92h9m49.235120866s) [noise=true , address=81.171.14.70:4042 , piece_id=WWJDEHBRMRVXANEWVG6B6QLD2SR7M6FXH4642UZ3AFQSDGCO2HYA, seg_pos={0 0}, upload=80368]: Uploading piece... Dec 9 11:38:49.364321191 (92h9m49.235404226s) [noise=true , address=81.171.14.70:4032 , piece_id=GOAKND7Z3IDBUYT3ILUUBU7YC3UMA6PPDJLLMFTH2OKHF76GCHWA, seg_pos={0 0}, upload=80368]: Uploading piece... Dec 9 11:38:49.364596545 (92h9m49.23567959s) [noise=true , address=81.171.14.70:4039 , piece_id=EJYQEP7LA6QJQICJQQVTNIXDN6ID5PAPL54GADSICUBJ3YQQ5E7Q, seg_pos={0 0}, upload=80368]: Uploading piece... Dec 9 11:38:49.365194575 (92h9m49.23627762s) [noise=true , address=89.149.201.147:4011 , piece_id=IJZKML6ZS5AF7KRYSHG4ZDR7AZSJGUWRSPTLSYTWR6WKYXWORAYA, seg_pos={0 0}, upload=80368]: Uploading piece... Dec 9 11:38:50.188365108 (92h9m50.059448153s) [upload=80368]: Next returned nil segment Dec 9 11:38:50.188379346 (92h9m50.059462391s) [upload=80368]: Waiting for error group managing segments... Dec 9 11:38:50.573198910 (92h9m50.444281955s) [noise=true , address=185.167.204.206:4062 , piece_id=UZUYTLORRUJOMEX62Z7ABCOXWZHBZ7PV72MA4TZIWDDWZKBR22OA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:50.577545046 (92h9m50.448628081s) [noise=true , address=185.167.204.206:4059 , piece_id=HLZEIABVICZTZCKXHBKIFQJ3UVRGVV5EWHWFFL5G3E7CYGJN4GHA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:50.577840359 (92h9m50.448923394s) [noise=true , address=185.167.204.206:4045 , piece_id=7RI3JKTQDSVERWTVQ4NIGZMBNAC6OPYUCTPU77HX3V74NBG7XALQ, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:50.578430443 (92h9m50.449513478s) [noise=true , address=185.167.204.206:4048 , piece_id=XMV7K5EEISKMT5JQFUD6EXZ3EYELHBWF6Y3J276VUUP2YGXJKF3A, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:50.579912414 (92h9m50.450995459s) [noise=true , address=185.167.204.205:4016 , piece_id=55J2Z5J7LPDPCBKXAQ7V4BQW4U7W6UAQVF63LYH3BPFHSEQE6CEA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:50.580088579 (92h9m50.451171625s) [noise=true , address=185.167.204.206:4068 , piece_id=FDF7EQNPFGNEDO6CEPYGMAAJOIT4TPHY2PVYHNXH7ILC7AY2LO3A, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:50.582475435 (92h9m50.45355848s) [noise=true , address=185.167.204.206:4037 , piece_id=LO332KWEY4NLBKVGEVME2SF7UIGTEVCKLOHRCC7FLIJEPP5DTKTA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:50.586259020 (92h9m50.457342055s) [noise=true , address=185.167.204.205:4030 , piece_id=VHBCZ6QQRBZ4THX6XCFIZ6JSFMIONQFSEZ3ERGOZB6L7L5USAKMA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:50.586458941 (92h9m50.457541986s) [noise=true , address=185.167.204.206:4060 , piece_id=A3NTK6DGBBOACMUONBGII27GVS7QZWOGDEKBPZFX76RSH5HMXS5A, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:50.588436907 (92h9m50.459519942s) [noise=true , address=185.167.204.205:4009 , piece_id=5WUDBEGUENMGEMSND2VCENBNGG2BIV47ICSWN3CJYLQQ3XIVZXKQ, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:50.589577038 (92h9m50.460660083s) [noise=true , address=185.167.204.205:4029 , piece_id=WMBEVUF62ZU76TQDICPDPMG4JA7374YQFXTI5XJFSN6MJZBATS7Q, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:50.592840462 (92h9m50.463923507s) [noise=true , address=185.167.204.205:4001 , piece_id=KIUIPM26OHXISSEHSZPLFHUXF4NVOHSY6TILREFVGDRDBB5IOCQQ, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.141501915 (92h9m51.01258496s) [noise=true , address=81.171.14.70:4039 , piece_id=EJYQEP7LA6QJQICJQQVTNIXDN6ID5PAPL54GADSICUBJ3YQQ5E7Q, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.153777871 (92h9m51.024860916s) [noise=true , address=81.171.14.70:4042 , piece_id=WWJDEHBRMRVXANEWVG6B6QLD2SR7M6FXH4642UZ3AFQSDGCO2HYA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.153948696 (92h9m51.025031751s) [noise=true , address=81.171.14.70:4028 , piece_id=NSYGSKJBIHMHZZX7HK7N3UQDGKWWQARJPGLAWH2TEPGWKNYWPC4A, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.157888027 (92h9m51.028971062s) [noise=true , address=81.171.14.70:4041 , piece_id=PZNQJLGVYH3CBPX4L2URCEVLMGK4JLQAMVFX25UBONKVYSTSEEWQ, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.170799434 (92h9m51.041882489s) [noise=true , address=81.171.14.70:4045 , piece_id=TZIZNHRR66JMWMZJNUHYFPYJ2ORR3OLKZ7VOQ5MIWXQOFARHL4VA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.171007360 (92h9m51.042090405s) [noise=true , address=81.171.14.70:4035 , piece_id=5T5MX4YDMI4IUY4LDSREN2MTN3NUL4GS6KZXDMNZMKJYY3VMS47Q, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.184999573 (92h9m51.056082648s) [noise=true , address=81.171.14.70:4032 , piece_id=GOAKND7Z3IDBUYT3ILUUBU7YC3UMA6PPDJLLMFTH2OKHF76GCHWA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.196460517 (92h9m51.067543562s) [noise=true , address=81.171.14.70:4040 , piece_id=4GA5CWKM5RJBL2IMWZNRGAA2T2JTFX54AFEFI3NAV2GNTKIDUELQ, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.199311115 (92h9m51.07039416s) [noise=true , address=81.171.14.70:4037 , piece_id=G6MBAEOMB7R4MQKZDULZY4DYQ5NR2QMCKW6HWYA55VD5YMSAERPA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.237519992 (92h9m51.108603028s) [noise=true , address=81.171.14.70:4030 , piece_id=VFAGZYZMOZKEHRNPUR2D2JCRDZFUFXUOS3JTTI2UOUYTIUY74NFQ, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.733107833 (92h9m51.604190868s) [noise=true , address=89.149.201.147:4005 , piece_id=OCHDE5VQIQPUVCOFXX343C7LXUIJFD4Q62ZZPEXKRN63KARUQR5A, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.815947736 (92h9m51.687030771s) [noise=true , address=89.149.201.147:4015 , piece_id=PGXJFJBHXVMXBTIWDTAOZKAU2JKTFSGCAD5I5MWU2MD32GF3AMDQ, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.841035780 (92h9m51.712118825s) [noise=true , address=89.149.201.147:4020 , piece_id=KIYFOE4WT3CGQQ3CL3BP2A3HH2KVKKLAUUGMHYPADVKI7BVBBRTA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.864720530 (92h9m51.735803575s) [noise=true , address=89.149.201.147:4002 , piece_id=TIK5YZA7N6MYGZ6KGNXPFBEQXD3DE2UXQ5Q2JP3EZGSLJ7TAO2GA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.864741660 (92h9m51.735824705s) [seg_pos={0 0}, upload=80368]: Segment reached optimal threshold of 26 pieces. Dec 9 11:38:51.865018136 (92h9m51.736101181s) [noise=true , address=89.149.201.147:4011 , piece_id=IJZKML6ZS5AF7KRYSHG4ZDR7AZSJGUWRSPTLSYTWR6WKYXWORAYA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PcxodnQMJERGjvBgnz3iEiC9MseRhMydkCriwGpQKqi6DGFDg): protocol: expected piece hash; context canceled; EOF Dec 9 11:38:51.865029489 (92h9m51.736112534s) [noise=true , address=89.149.201.147:4011 , piece_id=IJZKML6ZS5AF7KRYSHG4ZDR7AZSJGUWRSPTLSYTWR6WKYXWORAYA, seg_pos={0 0}, upload=80368]: Long tail cancellation due to optimal threshold Dec 9 11:38:51.865308640 (92h9m51.736391685s) [noise=true , address=89.149.201.147:4007 , piece_id=FHFZMTRZEYLGK2QOXICD5VLT77MIFP3HL6Z7SP7URQEY5L3HIDTA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Q7d4vj1KoWhDr6xd9gEfFrZ1sV3Yro2AFke2f3Ke1rXsGnvew): protocol: expected piece hash; context canceled; EOF Dec 9 11:38:51.865331703 (92h9m51.736414748s) [noise=true , address=89.149.201.147:4007 , piece_id=FHFZMTRZEYLGK2QOXICD5VLT77MIFP3HL6Z7SP7URQEY5L3HIDTA, seg_pos={0 0}, upload=80368]: Long tail cancellation due to optimal threshold Dec 9 11:38:51.865403281 (92h9m51.736486326s) [seg_pos={0 0}, upload=80368]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 9 11:38:51.865634440 (92h9m51.736717485s) [upload=80368]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 11:38:51.865643307 (92h9m51.736726352s) [upload=80368]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 11:40:37.870707645 (92h11m37.741790681s) [upload=80368]: Done uploading segments... Dec 9 11:40:37.870754294 (92h11m37.741837339s) [upload=80368]: Upload finished. err: metaclient: context deadline exceeded Duration: 18.169883336s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [.X............X....................XX................X.....X] duration: 5.000955671s trace: https://jaeger.dev.storj.io/trace/9c5f1cb27c256e3 FAIL (2025-12-09 12:02:00 UTC) orbiter warning: context deadline exceeded Duration: 5.000977762s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 8.627581651s trace: https://jaeger.dev.storj.io/trace/180c8a02a447001f FAIL (2025-12-09 12:01:53 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 1m24.245559s) [upload=79956]: Starting upload Dec 9 11:10:24.374550547 (91h41m24.245633592s) [upload=79956]: Done starting upload Dec 9 11:10:24.374695774 (91h41m24.245778819s) [upload=79956]: Uploading segments... Dec 9 11:10:24.374799832 (91h41m24.245882877s) [upload=79956]: Got next segment. Inline: false Dec 9 11:10:24.374845269 (91h41m24.245928314s) [upload=79956]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 11:10:24.374891817 (91h41m24.245974862s) [upload=79956]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 11:10:24.692953929 (91h41m24.564036964s) [seg_pos={0 0}, upload=79956]: Begin upload segment... Dec 9 11:10:24.692996710 (91h41m24.564079745s) [seg_pos={0 0}, upload=79956]: Done begin upload segment. Dec 9 11:10:24.693021047 (91h41m24.564104092s) [noise=true , address=145.79.227.42:4121 , piece_id=THP22HNINRQMLRTQJKBHRCSBE5O4JMZWVNFRTJDVFCP7CKP57V7Q, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:24.693030324 (91h41m24.56411336s) [noise=true , address=145.79.227.28:4727 , piece_id=2VL43MLR2PE4BPHW4IWKNWAW3PSOAX2EV32XWLQYMYBM4FZ7RT6Q, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:24.693620949 (91h41m24.564703994s) [noise=true , address=145.79.227.23:4545 , piece_id=XVIQ424I6HQC4IXH2XAYRG7DFXJ7LJ44VT7ZCWVHFFUTXMTBZYXQ, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:24.694151741 (91h41m24.565234786s) [noise=true , address=145.79.227.41:4087 , piece_id=CC643HDKTLAEAAOHWOSXJL75BJH62EL7QISOZVG2VYPH4S3LDKMQ, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:24.694508199 (91h41m24.565591244s) [noise=true , address=145.79.227.39:4014 , piece_id=4F3Y2MMBAZVYJW6T5ZRNLQ4ZU56GF4EQMY6BPKZTCJ7KR7EY6UXQ, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:24.694727136 (91h41m24.565810181s) [noise=true , address=45.91.116.4:4939 , piece_id=QK4MQXBPU7SZ7SUA7SMDF5IJ2UUA6Q6MA2MA4UR3DSMU6QN7OHSQ, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:24.695257175 (91h41m24.56634022s) [noise=true , address=145.79.227.26:4655 , piece_id=WSNSSYE2G6LWCKNBMS63KSEPZBQSENSZ536OHOEAHKMLAIVQA4ZA, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:24.695748691 (91h41m24.566831736s) [noise=true , address=145.79.227.27:4689 , piece_id=4PFOTYEIDSKPMQF4LFVZZB5MK4O6JXFTU2QCLC5GUZTAUFPOE45Q, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:24.696362650 (91h41m24.567445695s) [noise=true , address=145.79.227.31:4799 , piece_id=4GDZXDMIOOWAIEKGEVXHGKF2LQ4YLSCCPA4S6UPCCAVHGRUT4DCA, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:24.698217953 (91h41m24.569300988s) [noise=true , address=77.90.31.14:4661 , piece_id=LJY3BTAERCAOCCIKW2THHEBAYKOEXDYQVGJTMWHYBACZJ4XJTBBA, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:24.701427525 (91h41m24.57251057s) [noise=true , address=77.90.31.12:4592 , piece_id=THMPWJ2JV2RVPMFHR4UPYJEC2FT7AS7QYVZQI427KGYKTYGI3ZMQ, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:24.705981576 (91h41m24.577064621s) [noise=true , address=145.79.227.33:4869 , piece_id=R7N7EKAQ6PUL7IBCY7SH4GTGQ6XYU3ZB7UJUVKFX5S7NTFRSCFNA, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:24.708340077 (91h41m24.579423112s) [noise=true , address=145.79.227.27:4690 , piece_id=OBIAVWCVVSLUDUW7G7ZGIQRIVOHO5R4KR25AVMHNA4RSTT6KB23Q, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:24.710358000 (91h41m24.581441035s) [noise=true , address=145.79.227.43:4157 , piece_id=7WZ2NUCKMURKNDQCZ6FHUC2I4XFXK4W3B3DR26DK6F2G62R7YC7Q, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:24.711000353 (91h41m24.582083388s) [noise=true , address=145.79.227.29:4943 , piece_id=PVDPIUVHEL3DLK45G2NTR3INZJ2CEPKLF6KX6CZMOMKXEOFRTTBA, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:24.712392574 (91h41m24.583475609s) [noise=true , address=77.90.31.16:4734 , piece_id=BQYJNHPRU7WKEHXUXJCUX6ZDU2RSVDPAAJTOD3VEQCEGE7TVR6CQ, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:26.323507311 (91h41m26.194590346s) [upload=79956]: Next returned nil segment Dec 9 11:10:26.323527019 (91h41m26.194610064s) [upload=79956]: Waiting for error group managing segments... Dec 9 11:10:26.705924029 (91h41m26.577007074s) [noise=true , address=145.79.227.28:4727 , piece_id=2VL43MLR2PE4BPHW4IWKNWAW3PSOAX2EV32XWLQYMYBM4FZ7RT6Q, seg_pos={0 0}, upload=79956]: Done uploading piece. err: <nil> Dec 9 11:10:26.713527548 (91h41m26.584610593s) [noise=true , address=145.79.227.42:4121 , piece_id=THP22HNINRQMLRTQJKBHRCSBE5O4JMZWVNFRTJDVFCP7CKP57V7Q, seg_pos={0 0}, upload=79956]: Done uploading piece. err: <nil> Dec 9 11:10:26.715957385 (91h41m26.58704043s) [noise=true , address=145.79.227.23:4545 , piece_id=XVIQ424I6HQC4IXH2XAYRG7DFXJ7LJ44VT7ZCWVHFFUTXMTBZYXQ, seg_pos={0 0}, upload=79956]: Done uploading piece. err: <nil> Dec 9 11:10:26.721154390 (91h41m26.592237435s) [noise=true , address=145.79.227.26:4655 , piece_id=WSNSSYE2G6LWCKNBMS63KSEPZBQSENSZ536OHOEAHKMLAIVQA4ZA, seg_pos={0 0}, upload=79956]: Done uploading piece. err: <nil> Dec 9 11:10:26.726209116 (91h41m26.597292161s) [noise=true , address=145.79.227.43:4157 , piece_id=7WZ2NUCKMURKNDQCZ6FHUC2I4XFXK4W3B3DR26DK6F2G62R7YC7Q, seg_pos={0 0}, upload=79956]: Done uploading piece. err: <nil> Dec 9 11:10:26.744717479 (91h41m26.615800524s) [noise=true , address=145.79.227.31:4799 , piece_id=4GDZXDMIOOWAIEKGEVXHGKF2LQ4YLSCCPA4S6UPCCAVHGRUT4DCA, seg_pos={0 0}, upload=79956]: Done uploading piece. err: <nil> Dec 9 11:10:26.865540120 (91h41m26.736623165s) [noise=true , address=77.90.31.14:4661 , piece_id=LJY3BTAERCAOCCIKW2THHEBAYKOEXDYQVGJTMWHYBACZJ4XJTBBA, seg_pos={0 0}, upload=79956]: Done uploading piece. err: <nil> Dec 9 11:10:26.874733556 (91h41m26.745816591s) [noise=true , address=77.90.31.12:4592 , piece_id=THMPWJ2JV2RVPMFHR4UPYJEC2FT7AS7QYVZQI427KGYKTYGI3ZMQ, seg_pos={0 0}, upload=79956]: Done uploading piece. err: <nil> Dec 9 11:10:27.552332367 (91h41m27.423415412s) [noise=true , address=145.79.227.39:4014 , piece_id=4F3Y2MMBAZVYJW6T5ZRNLQ4ZU56GF4EQMY6BPKZTCJ7KR7EY6UXQ, seg_pos={0 0}, upload=79956]: Done uploading piece. err: <nil> Dec 9 11:10:27.555135495 (91h41m27.42621853s) [noise=true , address=145.79.227.27:4689 , piece_id=4PFOTYEIDSKPMQF4LFVZZB5MK4O6JXFTU2QCLC5GUZTAUFPOE45Q, seg_pos={0 0}, upload=79956]: Done uploading piece. err: <nil> Dec 9 11:10:27.837239356 (91h41m27.708322401s) [noise=true , address=145.79.227.41:4087 , piece_id=CC643HDKTLAEAAOHWOSXJL75BJH62EL7QISOZVG2VYPH4S3LDKMQ, seg_pos={0 0}, upload=79956]: Done uploading piece. err: <nil> Dec 9 11:10:27.850249129 (91h41m27.721332174s) [noise=true , address=145.79.227.29:4943 , piece_id=PVDPIUVHEL3DLK45G2NTR3INZJ2CEPKLF6KX6CZMOMKXEOFRTTBA, seg_pos={0 0}, upload=79956]: Done uploading piece. err: <nil> Dec 9 11:10:27.850441505 (91h41m27.72152455s) [noise=true , address=145.79.227.33:4869 , piece_id=R7N7EKAQ6PUL7IBCY7SH4GTGQ6XYU3ZB7UJUVKFX5S7NTFRSCFNA, seg_pos={0 0}, upload=79956]: Done uploading piece. err: <nil> Dec 9 11:10:27.850450944 (91h41m27.721533989s) [seg_pos={0 0}, upload=79956]: Segment reached optimal threshold of 13 pieces. Dec 9 11:10:27.850729945 (91h41m27.72181298s) [noise=true , address=77.90.31.16:4734 , piece_id=BQYJNHPRU7WKEHXUXJCUX6ZDU2RSVDPAAJTOD3VEQCEGE7TVR6CQ, seg_pos={0 0}, upload=79956]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: write tcp 188.166.225.170:57470->77.90.31.16:4734: use of closed network connection; write tcp 188.166.225.170:57470->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57470->77.90.31.16:4734: use of closed network connection Dec 9 11:10:27.850737028 (91h41m27.721820063s) [noise=true , address=77.90.31.16:4734 , piece_id=BQYJNHPRU7WKEHXUXJCUX6ZDU2RSVDPAAJTOD3VEQCEGE7TVR6CQ, seg_pos={0 0}, upload=79956]: Long tail cancellation due to optimal threshold Dec 9 11:10:27.850945425 (91h41m27.72202848s) [noise=true , address=45.91.116.4:4939 , piece_id=QK4MQXBPU7SZ7SUA7SMDF5IJ2UUA6Q6MA2MA4UR3DSMU6QN7OHSQ, seg_pos={0 0}, upload=79956]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: read tcp 188.166.225.170:48538->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 9 11:10:27.850974751 (91h41m27.722057816s) [noise=true , address=45.91.116.4:4939 , piece_id=QK4MQXBPU7SZ7SUA7SMDF5IJ2UUA6Q6MA2MA4UR3DSMU6QN7OHSQ, seg_pos={0 0}, upload=79956]: Long tail cancellation due to optimal threshold Dec 9 11:10:27.851168249 (91h41m27.722251294s) [noise=true , address=145.79.227.27:4690 , piece_id=OBIAVWCVVSLUDUW7G7ZGIQRIVOHO5R4KR25AVMHNA4RSTT6KB23Q, seg_pos={0 0}, upload=79956]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H): protocol: expected piece hash; context canceled; EOF Dec 9 11:10:27.851178119 (91h41m27.722261194s) [noise=true , address=145.79.227.27:4690 , piece_id=OBIAVWCVVSLUDUW7G7ZGIQRIVOHO5R4KR25AVMHNA4RSTT6KB23Q, seg_pos={0 0}, upload=79956]: Long tail cancellation due to optimal threshold Dec 9 11:10:27.851202003 (91h41m27.722285048s) [seg_pos={0 0}, upload=79956]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 9 11:10:27.851316833 (91h41m27.722399908s) [upload=79956]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 11:10:27.851325489 (91h41m27.722408534s) [upload=79956]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 11:25:29.822344337 (91h56m29.693427372s) [upload=79956]: Done uploading segments... Dec 9 11:25:29.822415463 (91h56m29.693498498s) [upload=79956]: Upload finished. err: metaclient: context deadline exceeded Duration: 8.627609164s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 20.288825875s trace: https://jaeger.dev.storj.io/trace/7098c007ca020436 FAIL (2025-12-09 12:01:48 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , address=185.167.204.205:4029 , piece_id=WMBEVUF62ZU76TQDICPDPMG4JA7374YQFXTI5XJFSN6MJZBATS7Q, seg_pos={0 0}, upload=80368]: Uploading piece... Dec 9 11:38:49.362641042 (92h9m49.233724077s) [noise=true , address=185.167.204.206:4037 , piece_id=LO332KWEY4NLBKVGEVME2SF7UIGTEVCKLOHRCC7FLIJEPP5DTKTA, seg_pos={0 0}, upload=80368]: Uploading piece... Dec 9 11:38:49.362747554 (92h9m49.233830599s) [noise=true , address=185.167.204.206:4045 , piece_id=7RI3JKTQDSVERWTVQ4NIGZMBNAC6OPYUCTPU77HX3V74NBG7XALQ, seg_pos={0 0}, upload=80368]: Uploading piece... Dec 9 11:38:49.363050582 (92h9m49.234133627s) [noise=true , address=185.167.204.206:4059 , piece_id=HLZEIABVICZTZCKXHBKIFQJ3UVRGVV5EWHWFFL5G3E7CYGJN4GHA, seg_pos={0 0}, upload=80368]: Uploading piece... Dec 9 11:38:49.363113251 (92h9m49.234196286s) [noise=true , address=185.167.204.205:4016 , piece_id=55J2Z5J7LPDPCBKXAQ7V4BQW4U7W6UAQVF63LYH3BPFHSEQE6CEA, seg_pos={0 0}, upload=80368]: Uploading piece... Dec 9 11:38:49.363383676 (92h9m49.234466711s) [noise=true , address=185.167.204.206:4048 , piece_id=XMV7K5EEISKMT5JQFUD6EXZ3EYELHBWF6Y3J276VUUP2YGXJKF3A, seg_pos={0 0}, upload=80368]: Uploading piece... Dec 9 11:38:49.363681573 (92h9m49.234764608s) [noise=true , address=185.167.204.205:4001 , piece_id=KIUIPM26OHXISSEHSZPLFHUXF4NVOHSY6TILREFVGDRDBB5IOCQQ, seg_pos={0 0}, upload=80368]: Uploading piece... Dec 9 11:38:49.364037821 (92h9m49.235120866s) [noise=true , address=81.171.14.70:4042 , piece_id=WWJDEHBRMRVXANEWVG6B6QLD2SR7M6FXH4642UZ3AFQSDGCO2HYA, seg_pos={0 0}, upload=80368]: Uploading piece... Dec 9 11:38:49.364321191 (92h9m49.235404226s) [noise=true , address=81.171.14.70:4032 , piece_id=GOAKND7Z3IDBUYT3ILUUBU7YC3UMA6PPDJLLMFTH2OKHF76GCHWA, seg_pos={0 0}, upload=80368]: Uploading piece... Dec 9 11:38:49.364596545 (92h9m49.23567959s) [noise=true , address=81.171.14.70:4039 , piece_id=EJYQEP7LA6QJQICJQQVTNIXDN6ID5PAPL54GADSICUBJ3YQQ5E7Q, seg_pos={0 0}, upload=80368]: Uploading piece... Dec 9 11:38:49.365194575 (92h9m49.23627762s) [noise=true , address=89.149.201.147:4011 , piece_id=IJZKML6ZS5AF7KRYSHG4ZDR7AZSJGUWRSPTLSYTWR6WKYXWORAYA, seg_pos={0 0}, upload=80368]: Uploading piece... Dec 9 11:38:50.188365108 (92h9m50.059448153s) [upload=80368]: Next returned nil segment Dec 9 11:38:50.188379346 (92h9m50.059462391s) [upload=80368]: Waiting for error group managing segments... Dec 9 11:38:50.573198910 (92h9m50.444281955s) [noise=true , address=185.167.204.206:4062 , piece_id=UZUYTLORRUJOMEX62Z7ABCOXWZHBZ7PV72MA4TZIWDDWZKBR22OA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:50.577545046 (92h9m50.448628081s) [noise=true , address=185.167.204.206:4059 , piece_id=HLZEIABVICZTZCKXHBKIFQJ3UVRGVV5EWHWFFL5G3E7CYGJN4GHA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:50.577840359 (92h9m50.448923394s) [noise=true , address=185.167.204.206:4045 , piece_id=7RI3JKTQDSVERWTVQ4NIGZMBNAC6OPYUCTPU77HX3V74NBG7XALQ, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:50.578430443 (92h9m50.449513478s) [noise=true , address=185.167.204.206:4048 , piece_id=XMV7K5EEISKMT5JQFUD6EXZ3EYELHBWF6Y3J276VUUP2YGXJKF3A, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:50.579912414 (92h9m50.450995459s) [noise=true , address=185.167.204.205:4016 , piece_id=55J2Z5J7LPDPCBKXAQ7V4BQW4U7W6UAQVF63LYH3BPFHSEQE6CEA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:50.580088579 (92h9m50.451171625s) [noise=true , address=185.167.204.206:4068 , piece_id=FDF7EQNPFGNEDO6CEPYGMAAJOIT4TPHY2PVYHNXH7ILC7AY2LO3A, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:50.582475435 (92h9m50.45355848s) [noise=true , address=185.167.204.206:4037 , piece_id=LO332KWEY4NLBKVGEVME2SF7UIGTEVCKLOHRCC7FLIJEPP5DTKTA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:50.586259020 (92h9m50.457342055s) [noise=true , address=185.167.204.205:4030 , piece_id=VHBCZ6QQRBZ4THX6XCFIZ6JSFMIONQFSEZ3ERGOZB6L7L5USAKMA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:50.586458941 (92h9m50.457541986s) [noise=true , address=185.167.204.206:4060 , piece_id=A3NTK6DGBBOACMUONBGII27GVS7QZWOGDEKBPZFX76RSH5HMXS5A, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:50.588436907 (92h9m50.459519942s) [noise=true , address=185.167.204.205:4009 , piece_id=5WUDBEGUENMGEMSND2VCENBNGG2BIV47ICSWN3CJYLQQ3XIVZXKQ, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:50.589577038 (92h9m50.460660083s) [noise=true , address=185.167.204.205:4029 , piece_id=WMBEVUF62ZU76TQDICPDPMG4JA7374YQFXTI5XJFSN6MJZBATS7Q, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:50.592840462 (92h9m50.463923507s) [noise=true , address=185.167.204.205:4001 , piece_id=KIUIPM26OHXISSEHSZPLFHUXF4NVOHSY6TILREFVGDRDBB5IOCQQ, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.141501915 (92h9m51.01258496s) [noise=true , address=81.171.14.70:4039 , piece_id=EJYQEP7LA6QJQICJQQVTNIXDN6ID5PAPL54GADSICUBJ3YQQ5E7Q, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.153777871 (92h9m51.024860916s) [noise=true , address=81.171.14.70:4042 , piece_id=WWJDEHBRMRVXANEWVG6B6QLD2SR7M6FXH4642UZ3AFQSDGCO2HYA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.153948696 (92h9m51.025031751s) [noise=true , address=81.171.14.70:4028 , piece_id=NSYGSKJBIHMHZZX7HK7N3UQDGKWWQARJPGLAWH2TEPGWKNYWPC4A, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.157888027 (92h9m51.028971062s) [noise=true , address=81.171.14.70:4041 , piece_id=PZNQJLGVYH3CBPX4L2URCEVLMGK4JLQAMVFX25UBONKVYSTSEEWQ, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.170799434 (92h9m51.041882489s) [noise=true , address=81.171.14.70:4045 , piece_id=TZIZNHRR66JMWMZJNUHYFPYJ2ORR3OLKZ7VOQ5MIWXQOFARHL4VA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.171007360 (92h9m51.042090405s) [noise=true , address=81.171.14.70:4035 , piece_id=5T5MX4YDMI4IUY4LDSREN2MTN3NUL4GS6KZXDMNZMKJYY3VMS47Q, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.184999573 (92h9m51.056082648s) [noise=true , address=81.171.14.70:4032 , piece_id=GOAKND7Z3IDBUYT3ILUUBU7YC3UMA6PPDJLLMFTH2OKHF76GCHWA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.196460517 (92h9m51.067543562s) [noise=true , address=81.171.14.70:4040 , piece_id=4GA5CWKM5RJBL2IMWZNRGAA2T2JTFX54AFEFI3NAV2GNTKIDUELQ, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.199311115 (92h9m51.07039416s) [noise=true , address=81.171.14.70:4037 , piece_id=G6MBAEOMB7R4MQKZDULZY4DYQ5NR2QMCKW6HWYA55VD5YMSAERPA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.237519992 (92h9m51.108603028s) [noise=true , address=81.171.14.70:4030 , piece_id=VFAGZYZMOZKEHRNPUR2D2JCRDZFUFXUOS3JTTI2UOUYTIUY74NFQ, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.733107833 (92h9m51.604190868s) [noise=true , address=89.149.201.147:4005 , piece_id=OCHDE5VQIQPUVCOFXX343C7LXUIJFD4Q62ZZPEXKRN63KARUQR5A, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.815947736 (92h9m51.687030771s) [noise=true , address=89.149.201.147:4015 , piece_id=PGXJFJBHXVMXBTIWDTAOZKAU2JKTFSGCAD5I5MWU2MD32GF3AMDQ, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.841035780 (92h9m51.712118825s) [noise=true , address=89.149.201.147:4020 , piece_id=KIYFOE4WT3CGQQ3CL3BP2A3HH2KVKKLAUUGMHYPADVKI7BVBBRTA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.864720530 (92h9m51.735803575s) [noise=true , address=89.149.201.147:4002 , piece_id=TIK5YZA7N6MYGZ6KGNXPFBEQXD3DE2UXQ5Q2JP3EZGSLJ7TAO2GA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.864741660 (92h9m51.735824705s) [seg_pos={0 0}, upload=80368]: Segment reached optimal threshold of 26 pieces. Dec 9 11:38:51.865018136 (92h9m51.736101181s) [noise=true , address=89.149.201.147:4011 , piece_id=IJZKML6ZS5AF7KRYSHG4ZDR7AZSJGUWRSPTLSYTWR6WKYXWORAYA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PcxodnQMJERGjvBgnz3iEiC9MseRhMydkCriwGpQKqi6DGFDg): protocol: expected piece hash; context canceled; EOF Dec 9 11:38:51.865029489 (92h9m51.736112534s) [noise=true , address=89.149.201.147:4011 , piece_id=IJZKML6ZS5AF7KRYSHG4ZDR7AZSJGUWRSPTLSYTWR6WKYXWORAYA, seg_pos={0 0}, upload=80368]: Long tail cancellation due to optimal threshold Dec 9 11:38:51.865308640 (92h9m51.736391685s) [noise=true , address=89.149.201.147:4007 , piece_id=FHFZMTRZEYLGK2QOXICD5VLT77MIFP3HL6Z7SP7URQEY5L3HIDTA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Q7d4vj1KoWhDr6xd9gEfFrZ1sV3Yro2AFke2f3Ke1rXsGnvew): protocol: expected piece hash; context canceled; EOF Dec 9 11:38:51.865331703 (92h9m51.736414748s) [noise=true , address=89.149.201.147:4007 , piece_id=FHFZMTRZEYLGK2QOXICD5VLT77MIFP3HL6Z7SP7URQEY5L3HIDTA, seg_pos={0 0}, upload=80368]: Long tail cancellation due to optimal threshold Dec 9 11:38:51.865403281 (92h9m51.736486326s) [seg_pos={0 0}, upload=80368]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 9 11:38:51.865634440 (92h9m51.736717485s) [upload=80368]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 11:38:51.865643307 (92h9m51.736726352s) [upload=80368]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 11:40:37.870707645 (92h11m37.741790681s) [upload=80368]: Done uploading segments... Dec 9 11:40:37.870754294 (92h11m37.741837339s) [upload=80368]: Upload finished. err: metaclient: context deadline exceeded Duration: 20.288858577s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 5.958310923s trace: https://jaeger.dev.storj.io/trace/1ad10e1389335c60 FAIL (2025-12-09 12:01:44 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.958343346s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 5 out of 60 is failed (threshold=6) [..X............X....................XX................X.....] duration: 3.60989271s trace: https://jaeger.dev.storj.io/trace/36e1a86555e8f5a8 FAIL (2025-12-09 12:01:43 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.609919121s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 476.756432ms remote: "tcp:35.207.121.91:443" target: https://eu1.storj.io/login trace: https://jaeger.dev.storj.io/trace/215c1ddb7fa82bd3 FAIL (2025-12-09 12:01:42 UTC) orbiter warning: Get "https://eu1.storj.io/login": EOF Duration: 476.777843ms core satellite eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 5.794767881s trace: https://jaeger.dev.storj.io/trace/1ad10e1389335c60 FAIL (2025-12-09 12:01:21 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.794797497s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 466.750227ms remote: "tcp:35.207.121.91:443" target: https://eu1.storj.io/login trace: https://jaeger.dev.storj.io/trace/215c1ddb7fa82bd3 FAIL (2025-12-09 12:01:20 UTC) orbiter warning: Get "https://eu1.storj.io/login": EOF Duration: 466.77882ms core satellite eu1
checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 469.442563ms remote: "tcp:35.207.121.91:443" target: https://eu1.storj.io/login trace: https://jaeger.dev.storj.io/trace/215c1ddb7fa82bd3 FAIL (2025-12-09 12:00:59 UTC) orbiter warning: Get "https://eu1.storj.io/login": EOF Duration: 469.489291ms core satellite eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [X..X............X....................XX................X....] duration: 4.336771388s trace: https://jaeger.dev.storj.io/trace/36e1a86555e8f5a8 FAIL (2025-12-09 12:00:54 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.336808541s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 14.294987582s trace: https://jaeger.dev.storj.io/trace/180c8a02a447001f FAIL (2025-12-09 12:00:48 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 1m24.245559s) [upload=79956]: Starting upload Dec 9 11:10:24.374550547 (91h41m24.245633592s) [upload=79956]: Done starting upload Dec 9 11:10:24.374695774 (91h41m24.245778819s) [upload=79956]: Uploading segments... Dec 9 11:10:24.374799832 (91h41m24.245882877s) [upload=79956]: Got next segment. Inline: false Dec 9 11:10:24.374845269 (91h41m24.245928314s) [upload=79956]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 11:10:24.374891817 (91h41m24.245974862s) [upload=79956]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 11:10:24.692953929 (91h41m24.564036964s) [seg_pos={0 0}, upload=79956]: Begin upload segment... Dec 9 11:10:24.692996710 (91h41m24.564079745s) [seg_pos={0 0}, upload=79956]: Done begin upload segment. Dec 9 11:10:24.693021047 (91h41m24.564104092s) [noise=true , address=145.79.227.42:4121 , piece_id=THP22HNINRQMLRTQJKBHRCSBE5O4JMZWVNFRTJDVFCP7CKP57V7Q, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:24.693030324 (91h41m24.56411336s) [noise=true , address=145.79.227.28:4727 , piece_id=2VL43MLR2PE4BPHW4IWKNWAW3PSOAX2EV32XWLQYMYBM4FZ7RT6Q, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:24.693620949 (91h41m24.564703994s) [noise=true , address=145.79.227.23:4545 , piece_id=XVIQ424I6HQC4IXH2XAYRG7DFXJ7LJ44VT7ZCWVHFFUTXMTBZYXQ, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:24.694151741 (91h41m24.565234786s) [noise=true , address=145.79.227.41:4087 , piece_id=CC643HDKTLAEAAOHWOSXJL75BJH62EL7QISOZVG2VYPH4S3LDKMQ, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:24.694508199 (91h41m24.565591244s) [noise=true , address=145.79.227.39:4014 , piece_id=4F3Y2MMBAZVYJW6T5ZRNLQ4ZU56GF4EQMY6BPKZTCJ7KR7EY6UXQ, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:24.694727136 (91h41m24.565810181s) [noise=true , address=45.91.116.4:4939 , piece_id=QK4MQXBPU7SZ7SUA7SMDF5IJ2UUA6Q6MA2MA4UR3DSMU6QN7OHSQ, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:24.695257175 (91h41m24.56634022s) [noise=true , address=145.79.227.26:4655 , piece_id=WSNSSYE2G6LWCKNBMS63KSEPZBQSENSZ536OHOEAHKMLAIVQA4ZA, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:24.695748691 (91h41m24.566831736s) [noise=true , address=145.79.227.27:4689 , piece_id=4PFOTYEIDSKPMQF4LFVZZB5MK4O6JXFTU2QCLC5GUZTAUFPOE45Q, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:24.696362650 (91h41m24.567445695s) [noise=true , address=145.79.227.31:4799 , piece_id=4GDZXDMIOOWAIEKGEVXHGKF2LQ4YLSCCPA4S6UPCCAVHGRUT4DCA, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:24.698217953 (91h41m24.569300988s) [noise=true , address=77.90.31.14:4661 , piece_id=LJY3BTAERCAOCCIKW2THHEBAYKOEXDYQVGJTMWHYBACZJ4XJTBBA, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:24.701427525 (91h41m24.57251057s) [noise=true , address=77.90.31.12:4592 , piece_id=THMPWJ2JV2RVPMFHR4UPYJEC2FT7AS7QYVZQI427KGYKTYGI3ZMQ, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:24.705981576 (91h41m24.577064621s) [noise=true , address=145.79.227.33:4869 , piece_id=R7N7EKAQ6PUL7IBCY7SH4GTGQ6XYU3ZB7UJUVKFX5S7NTFRSCFNA, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:24.708340077 (91h41m24.579423112s) [noise=true , address=145.79.227.27:4690 , piece_id=OBIAVWCVVSLUDUW7G7ZGIQRIVOHO5R4KR25AVMHNA4RSTT6KB23Q, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:24.710358000 (91h41m24.581441035s) [noise=true , address=145.79.227.43:4157 , piece_id=7WZ2NUCKMURKNDQCZ6FHUC2I4XFXK4W3B3DR26DK6F2G62R7YC7Q, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:24.711000353 (91h41m24.582083388s) [noise=true , address=145.79.227.29:4943 , piece_id=PVDPIUVHEL3DLK45G2NTR3INZJ2CEPKLF6KX6CZMOMKXEOFRTTBA, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:24.712392574 (91h41m24.583475609s) [noise=true , address=77.90.31.16:4734 , piece_id=BQYJNHPRU7WKEHXUXJCUX6ZDU2RSVDPAAJTOD3VEQCEGE7TVR6CQ, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:26.323507311 (91h41m26.194590346s) [upload=79956]: Next returned nil segment Dec 9 11:10:26.323527019 (91h41m26.194610064s) [upload=79956]: Waiting for error group managing segments... Dec 9 11:10:26.705924029 (91h41m26.577007074s) [noise=true , address=145.79.227.28:4727 , piece_id=2VL43MLR2PE4BPHW4IWKNWAW3PSOAX2EV32XWLQYMYBM4FZ7RT6Q, seg_pos={0 0}, upload=79956]: Done uploading piece. err: <nil> Dec 9 11:10:26.713527548 (91h41m26.584610593s) [noise=true , address=145.79.227.42:4121 , piece_id=THP22HNINRQMLRTQJKBHRCSBE5O4JMZWVNFRTJDVFCP7CKP57V7Q, seg_pos={0 0}, upload=79956]: Done uploading piece. err: <nil> Dec 9 11:10:26.715957385 (91h41m26.58704043s) [noise=true , address=145.79.227.23:4545 , piece_id=XVIQ424I6HQC4IXH2XAYRG7DFXJ7LJ44VT7ZCWVHFFUTXMTBZYXQ, seg_pos={0 0}, upload=79956]: Done uploading piece. err: <nil> Dec 9 11:10:26.721154390 (91h41m26.592237435s) [noise=true , address=145.79.227.26:4655 , piece_id=WSNSSYE2G6LWCKNBMS63KSEPZBQSENSZ536OHOEAHKMLAIVQA4ZA, seg_pos={0 0}, upload=79956]: Done uploading piece. err: <nil> Dec 9 11:10:26.726209116 (91h41m26.597292161s) [noise=true , address=145.79.227.43:4157 , piece_id=7WZ2NUCKMURKNDQCZ6FHUC2I4XFXK4W3B3DR26DK6F2G62R7YC7Q, seg_pos={0 0}, upload=79956]: Done uploading piece. err: <nil> Dec 9 11:10:26.744717479 (91h41m26.615800524s) [noise=true , address=145.79.227.31:4799 , piece_id=4GDZXDMIOOWAIEKGEVXHGKF2LQ4YLSCCPA4S6UPCCAVHGRUT4DCA, seg_pos={0 0}, upload=79956]: Done uploading piece. err: <nil> Dec 9 11:10:26.865540120 (91h41m26.736623165s) [noise=true , address=77.90.31.14:4661 , piece_id=LJY3BTAERCAOCCIKW2THHEBAYKOEXDYQVGJTMWHYBACZJ4XJTBBA, seg_pos={0 0}, upload=79956]: Done uploading piece. err: <nil> Dec 9 11:10:26.874733556 (91h41m26.745816591s) [noise=true , address=77.90.31.12:4592 , piece_id=THMPWJ2JV2RVPMFHR4UPYJEC2FT7AS7QYVZQI427KGYKTYGI3ZMQ, seg_pos={0 0}, upload=79956]: Done uploading piece. err: <nil> Dec 9 11:10:27.552332367 (91h41m27.423415412s) [noise=true , address=145.79.227.39:4014 , piece_id=4F3Y2MMBAZVYJW6T5ZRNLQ4ZU56GF4EQMY6BPKZTCJ7KR7EY6UXQ, seg_pos={0 0}, upload=79956]: Done uploading piece. err: <nil> Dec 9 11:10:27.555135495 (91h41m27.42621853s) [noise=true , address=145.79.227.27:4689 , piece_id=4PFOTYEIDSKPMQF4LFVZZB5MK4O6JXFTU2QCLC5GUZTAUFPOE45Q, seg_pos={0 0}, upload=79956]: Done uploading piece. err: <nil> Dec 9 11:10:27.837239356 (91h41m27.708322401s) [noise=true , address=145.79.227.41:4087 , piece_id=CC643HDKTLAEAAOHWOSXJL75BJH62EL7QISOZVG2VYPH4S3LDKMQ, seg_pos={0 0}, upload=79956]: Done uploading piece. err: <nil> Dec 9 11:10:27.850249129 (91h41m27.721332174s) [noise=true , address=145.79.227.29:4943 , piece_id=PVDPIUVHEL3DLK45G2NTR3INZJ2CEPKLF6KX6CZMOMKXEOFRTTBA, seg_pos={0 0}, upload=79956]: Done uploading piece. err: <nil> Dec 9 11:10:27.850441505 (91h41m27.72152455s) [noise=true , address=145.79.227.33:4869 , piece_id=R7N7EKAQ6PUL7IBCY7SH4GTGQ6XYU3ZB7UJUVKFX5S7NTFRSCFNA, seg_pos={0 0}, upload=79956]: Done uploading piece. err: <nil> Dec 9 11:10:27.850450944 (91h41m27.721533989s) [seg_pos={0 0}, upload=79956]: Segment reached optimal threshold of 13 pieces. Dec 9 11:10:27.850729945 (91h41m27.72181298s) [noise=true , address=77.90.31.16:4734 , piece_id=BQYJNHPRU7WKEHXUXJCUX6ZDU2RSVDPAAJTOD3VEQCEGE7TVR6CQ, seg_pos={0 0}, upload=79956]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: write tcp 188.166.225.170:57470->77.90.31.16:4734: use of closed network connection; write tcp 188.166.225.170:57470->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57470->77.90.31.16:4734: use of closed network connection Dec 9 11:10:27.850737028 (91h41m27.721820063s) [noise=true , address=77.90.31.16:4734 , piece_id=BQYJNHPRU7WKEHXUXJCUX6ZDU2RSVDPAAJTOD3VEQCEGE7TVR6CQ, seg_pos={0 0}, upload=79956]: Long tail cancellation due to optimal threshold Dec 9 11:10:27.850945425 (91h41m27.72202848s) [noise=true , address=45.91.116.4:4939 , piece_id=QK4MQXBPU7SZ7SUA7SMDF5IJ2UUA6Q6MA2MA4UR3DSMU6QN7OHSQ, seg_pos={0 0}, upload=79956]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: read tcp 188.166.225.170:48538->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 9 11:10:27.850974751 (91h41m27.722057816s) [noise=true , address=45.91.116.4:4939 , piece_id=QK4MQXBPU7SZ7SUA7SMDF5IJ2UUA6Q6MA2MA4UR3DSMU6QN7OHSQ, seg_pos={0 0}, upload=79956]: Long tail cancellation due to optimal threshold Dec 9 11:10:27.851168249 (91h41m27.722251294s) [noise=true , address=145.79.227.27:4690 , piece_id=OBIAVWCVVSLUDUW7G7ZGIQRIVOHO5R4KR25AVMHNA4RSTT6KB23Q, seg_pos={0 0}, upload=79956]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H): protocol: expected piece hash; context canceled; EOF Dec 9 11:10:27.851178119 (91h41m27.722261194s) [noise=true , address=145.79.227.27:4690 , piece_id=OBIAVWCVVSLUDUW7G7ZGIQRIVOHO5R4KR25AVMHNA4RSTT6KB23Q, seg_pos={0 0}, upload=79956]: Long tail cancellation due to optimal threshold Dec 9 11:10:27.851202003 (91h41m27.722285048s) [seg_pos={0 0}, upload=79956]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 9 11:10:27.851316833 (91h41m27.722399908s) [upload=79956]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 11:10:27.851325489 (91h41m27.722408534s) [upload=79956]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 11:25:29.822344337 (91h56m29.693427372s) [upload=79956]: Done uploading segments... Dec 9 11:25:29.822415463 (91h56m29.693498498s) [upload=79956]: Upload finished. err: metaclient: context deadline exceeded Duration: 14.295016407s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 15.746970434s trace: https://jaeger.dev.storj.io/trace/7098c007ca020436 FAIL (2025-12-09 12:00:35 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , address=185.167.204.205:4029 , piece_id=WMBEVUF62ZU76TQDICPDPMG4JA7374YQFXTI5XJFSN6MJZBATS7Q, seg_pos={0 0}, upload=80368]: Uploading piece... Dec 9 11:38:49.362641042 (92h9m49.233724077s) [noise=true , address=185.167.204.206:4037 , piece_id=LO332KWEY4NLBKVGEVME2SF7UIGTEVCKLOHRCC7FLIJEPP5DTKTA, seg_pos={0 0}, upload=80368]: Uploading piece... Dec 9 11:38:49.362747554 (92h9m49.233830599s) [noise=true , address=185.167.204.206:4045 , piece_id=7RI3JKTQDSVERWTVQ4NIGZMBNAC6OPYUCTPU77HX3V74NBG7XALQ, seg_pos={0 0}, upload=80368]: Uploading piece... Dec 9 11:38:49.363050582 (92h9m49.234133627s) [noise=true , address=185.167.204.206:4059 , piece_id=HLZEIABVICZTZCKXHBKIFQJ3UVRGVV5EWHWFFL5G3E7CYGJN4GHA, seg_pos={0 0}, upload=80368]: Uploading piece... Dec 9 11:38:49.363113251 (92h9m49.234196286s) [noise=true , address=185.167.204.205:4016 , piece_id=55J2Z5J7LPDPCBKXAQ7V4BQW4U7W6UAQVF63LYH3BPFHSEQE6CEA, seg_pos={0 0}, upload=80368]: Uploading piece... Dec 9 11:38:49.363383676 (92h9m49.234466711s) [noise=true , address=185.167.204.206:4048 , piece_id=XMV7K5EEISKMT5JQFUD6EXZ3EYELHBWF6Y3J276VUUP2YGXJKF3A, seg_pos={0 0}, upload=80368]: Uploading piece... Dec 9 11:38:49.363681573 (92h9m49.234764608s) [noise=true , address=185.167.204.205:4001 , piece_id=KIUIPM26OHXISSEHSZPLFHUXF4NVOHSY6TILREFVGDRDBB5IOCQQ, seg_pos={0 0}, upload=80368]: Uploading piece... Dec 9 11:38:49.364037821 (92h9m49.235120866s) [noise=true , address=81.171.14.70:4042 , piece_id=WWJDEHBRMRVXANEWVG6B6QLD2SR7M6FXH4642UZ3AFQSDGCO2HYA, seg_pos={0 0}, upload=80368]: Uploading piece... Dec 9 11:38:49.364321191 (92h9m49.235404226s) [noise=true , address=81.171.14.70:4032 , piece_id=GOAKND7Z3IDBUYT3ILUUBU7YC3UMA6PPDJLLMFTH2OKHF76GCHWA, seg_pos={0 0}, upload=80368]: Uploading piece... Dec 9 11:38:49.364596545 (92h9m49.23567959s) [noise=true , address=81.171.14.70:4039 , piece_id=EJYQEP7LA6QJQICJQQVTNIXDN6ID5PAPL54GADSICUBJ3YQQ5E7Q, seg_pos={0 0}, upload=80368]: Uploading piece... Dec 9 11:38:49.365194575 (92h9m49.23627762s) [noise=true , address=89.149.201.147:4011 , piece_id=IJZKML6ZS5AF7KRYSHG4ZDR7AZSJGUWRSPTLSYTWR6WKYXWORAYA, seg_pos={0 0}, upload=80368]: Uploading piece... Dec 9 11:38:50.188365108 (92h9m50.059448153s) [upload=80368]: Next returned nil segment Dec 9 11:38:50.188379346 (92h9m50.059462391s) [upload=80368]: Waiting for error group managing segments... Dec 9 11:38:50.573198910 (92h9m50.444281955s) [noise=true , address=185.167.204.206:4062 , piece_id=UZUYTLORRUJOMEX62Z7ABCOXWZHBZ7PV72MA4TZIWDDWZKBR22OA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:50.577545046 (92h9m50.448628081s) [noise=true , address=185.167.204.206:4059 , piece_id=HLZEIABVICZTZCKXHBKIFQJ3UVRGVV5EWHWFFL5G3E7CYGJN4GHA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:50.577840359 (92h9m50.448923394s) [noise=true , address=185.167.204.206:4045 , piece_id=7RI3JKTQDSVERWTVQ4NIGZMBNAC6OPYUCTPU77HX3V74NBG7XALQ, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:50.578430443 (92h9m50.449513478s) [noise=true , address=185.167.204.206:4048 , piece_id=XMV7K5EEISKMT5JQFUD6EXZ3EYELHBWF6Y3J276VUUP2YGXJKF3A, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:50.579912414 (92h9m50.450995459s) [noise=true , address=185.167.204.205:4016 , piece_id=55J2Z5J7LPDPCBKXAQ7V4BQW4U7W6UAQVF63LYH3BPFHSEQE6CEA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:50.580088579 (92h9m50.451171625s) [noise=true , address=185.167.204.206:4068 , piece_id=FDF7EQNPFGNEDO6CEPYGMAAJOIT4TPHY2PVYHNXH7ILC7AY2LO3A, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:50.582475435 (92h9m50.45355848s) [noise=true , address=185.167.204.206:4037 , piece_id=LO332KWEY4NLBKVGEVME2SF7UIGTEVCKLOHRCC7FLIJEPP5DTKTA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:50.586259020 (92h9m50.457342055s) [noise=true , address=185.167.204.205:4030 , piece_id=VHBCZ6QQRBZ4THX6XCFIZ6JSFMIONQFSEZ3ERGOZB6L7L5USAKMA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:50.586458941 (92h9m50.457541986s) [noise=true , address=185.167.204.206:4060 , piece_id=A3NTK6DGBBOACMUONBGII27GVS7QZWOGDEKBPZFX76RSH5HMXS5A, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:50.588436907 (92h9m50.459519942s) [noise=true , address=185.167.204.205:4009 , piece_id=5WUDBEGUENMGEMSND2VCENBNGG2BIV47ICSWN3CJYLQQ3XIVZXKQ, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:50.589577038 (92h9m50.460660083s) [noise=true , address=185.167.204.205:4029 , piece_id=WMBEVUF62ZU76TQDICPDPMG4JA7374YQFXTI5XJFSN6MJZBATS7Q, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:50.592840462 (92h9m50.463923507s) [noise=true , address=185.167.204.205:4001 , piece_id=KIUIPM26OHXISSEHSZPLFHUXF4NVOHSY6TILREFVGDRDBB5IOCQQ, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.141501915 (92h9m51.01258496s) [noise=true , address=81.171.14.70:4039 , piece_id=EJYQEP7LA6QJQICJQQVTNIXDN6ID5PAPL54GADSICUBJ3YQQ5E7Q, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.153777871 (92h9m51.024860916s) [noise=true , address=81.171.14.70:4042 , piece_id=WWJDEHBRMRVXANEWVG6B6QLD2SR7M6FXH4642UZ3AFQSDGCO2HYA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.153948696 (92h9m51.025031751s) [noise=true , address=81.171.14.70:4028 , piece_id=NSYGSKJBIHMHZZX7HK7N3UQDGKWWQARJPGLAWH2TEPGWKNYWPC4A, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.157888027 (92h9m51.028971062s) [noise=true , address=81.171.14.70:4041 , piece_id=PZNQJLGVYH3CBPX4L2URCEVLMGK4JLQAMVFX25UBONKVYSTSEEWQ, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.170799434 (92h9m51.041882489s) [noise=true , address=81.171.14.70:4045 , piece_id=TZIZNHRR66JMWMZJNUHYFPYJ2ORR3OLKZ7VOQ5MIWXQOFARHL4VA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.171007360 (92h9m51.042090405s) [noise=true , address=81.171.14.70:4035 , piece_id=5T5MX4YDMI4IUY4LDSREN2MTN3NUL4GS6KZXDMNZMKJYY3VMS47Q, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.184999573 (92h9m51.056082648s) [noise=true , address=81.171.14.70:4032 , piece_id=GOAKND7Z3IDBUYT3ILUUBU7YC3UMA6PPDJLLMFTH2OKHF76GCHWA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.196460517 (92h9m51.067543562s) [noise=true , address=81.171.14.70:4040 , piece_id=4GA5CWKM5RJBL2IMWZNRGAA2T2JTFX54AFEFI3NAV2GNTKIDUELQ, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.199311115 (92h9m51.07039416s) [noise=true , address=81.171.14.70:4037 , piece_id=G6MBAEOMB7R4MQKZDULZY4DYQ5NR2QMCKW6HWYA55VD5YMSAERPA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.237519992 (92h9m51.108603028s) [noise=true , address=81.171.14.70:4030 , piece_id=VFAGZYZMOZKEHRNPUR2D2JCRDZFUFXUOS3JTTI2UOUYTIUY74NFQ, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.733107833 (92h9m51.604190868s) [noise=true , address=89.149.201.147:4005 , piece_id=OCHDE5VQIQPUVCOFXX343C7LXUIJFD4Q62ZZPEXKRN63KARUQR5A, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.815947736 (92h9m51.687030771s) [noise=true , address=89.149.201.147:4015 , piece_id=PGXJFJBHXVMXBTIWDTAOZKAU2JKTFSGCAD5I5MWU2MD32GF3AMDQ, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.841035780 (92h9m51.712118825s) [noise=true , address=89.149.201.147:4020 , piece_id=KIYFOE4WT3CGQQ3CL3BP2A3HH2KVKKLAUUGMHYPADVKI7BVBBRTA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.864720530 (92h9m51.735803575s) [noise=true , address=89.149.201.147:4002 , piece_id=TIK5YZA7N6MYGZ6KGNXPFBEQXD3DE2UXQ5Q2JP3EZGSLJ7TAO2GA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.864741660 (92h9m51.735824705s) [seg_pos={0 0}, upload=80368]: Segment reached optimal threshold of 26 pieces. Dec 9 11:38:51.865018136 (92h9m51.736101181s) [noise=true , address=89.149.201.147:4011 , piece_id=IJZKML6ZS5AF7KRYSHG4ZDR7AZSJGUWRSPTLSYTWR6WKYXWORAYA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PcxodnQMJERGjvBgnz3iEiC9MseRhMydkCriwGpQKqi6DGFDg): protocol: expected piece hash; context canceled; EOF Dec 9 11:38:51.865029489 (92h9m51.736112534s) [noise=true , address=89.149.201.147:4011 , piece_id=IJZKML6ZS5AF7KRYSHG4ZDR7AZSJGUWRSPTLSYTWR6WKYXWORAYA, seg_pos={0 0}, upload=80368]: Long tail cancellation due to optimal threshold Dec 9 11:38:51.865308640 (92h9m51.736391685s) [noise=true , address=89.149.201.147:4007 , piece_id=FHFZMTRZEYLGK2QOXICD5VLT77MIFP3HL6Z7SP7URQEY5L3HIDTA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Q7d4vj1KoWhDr6xd9gEfFrZ1sV3Yro2AFke2f3Ke1rXsGnvew): protocol: expected piece hash; context canceled; EOF Dec 9 11:38:51.865331703 (92h9m51.736414748s) [noise=true , address=89.149.201.147:4007 , piece_id=FHFZMTRZEYLGK2QOXICD5VLT77MIFP3HL6Z7SP7URQEY5L3HIDTA, seg_pos={0 0}, upload=80368]: Long tail cancellation due to optimal threshold Dec 9 11:38:51.865403281 (92h9m51.736486326s) [seg_pos={0 0}, upload=80368]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 9 11:38:51.865634440 (92h9m51.736717485s) [upload=80368]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 11:38:51.865643307 (92h9m51.736726352s) [upload=80368]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 11:40:37.870707645 (92h11m37.741790681s) [upload=80368]: Done uploading segments... Dec 9 11:40:37.870754294 (92h11m37.741837339s) [upload=80368]: Upload finished. err: metaclient: context deadline exceeded Duration: 15.747002235s core uplink eu-select-1 eu1-placement-15
checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 476.528103ms remote: "tcp:35.207.121.91:443" target: https://eu1.storj.io/login trace: https://jaeger.dev.storj.io/trace/215c1ddb7fa82bd3 FAIL (2025-12-09 12:00:28 UTC) orbiter warning: Get "https://eu1.storj.io/login": EOF Duration: 476.551568ms core satellite eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 5.935848397s trace: https://jaeger.dev.storj.io/trace/1ad10e1389335c60 FAIL (2025-12-09 12:00:25 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.935869937s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 465.593472ms remote: "tcp:35.207.121.91:443" target: https://eu1.storj.io/login trace: https://jaeger.dev.storj.io/trace/215c1ddb7fa82bd3 FAIL (2025-12-09 11:59:57 UTC) orbiter warning: Get "https://eu1.storj.io/login": EOF Duration: 465.620343ms core satellite eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 5.658655329s trace: https://jaeger.dev.storj.io/trace/1ad10e1389335c60 FAIL (2025-12-09 11:59:56 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.658706165s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [.X..X............X....................XX................X...] duration: 3.507900198s trace: https://jaeger.dev.storj.io/trace/36e1a86555e8f5a8 FAIL (2025-12-09 11:59:50 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.507935485s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 13.018089632s trace: https://jaeger.dev.storj.io/trace/180c8a02a447001f FAIL (2025-12-09 11:59:48 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 1m24.245559s) [upload=79956]: Starting upload Dec 9 11:10:24.374550547 (91h41m24.245633592s) [upload=79956]: Done starting upload Dec 9 11:10:24.374695774 (91h41m24.245778819s) [upload=79956]: Uploading segments... Dec 9 11:10:24.374799832 (91h41m24.245882877s) [upload=79956]: Got next segment. Inline: false Dec 9 11:10:24.374845269 (91h41m24.245928314s) [upload=79956]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 11:10:24.374891817 (91h41m24.245974862s) [upload=79956]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 11:10:24.692953929 (91h41m24.564036964s) [seg_pos={0 0}, upload=79956]: Begin upload segment... Dec 9 11:10:24.692996710 (91h41m24.564079745s) [seg_pos={0 0}, upload=79956]: Done begin upload segment. Dec 9 11:10:24.693021047 (91h41m24.564104092s) [noise=true , address=145.79.227.42:4121 , piece_id=THP22HNINRQMLRTQJKBHRCSBE5O4JMZWVNFRTJDVFCP7CKP57V7Q, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:24.693030324 (91h41m24.56411336s) [noise=true , address=145.79.227.28:4727 , piece_id=2VL43MLR2PE4BPHW4IWKNWAW3PSOAX2EV32XWLQYMYBM4FZ7RT6Q, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:24.693620949 (91h41m24.564703994s) [noise=true , address=145.79.227.23:4545 , piece_id=XVIQ424I6HQC4IXH2XAYRG7DFXJ7LJ44VT7ZCWVHFFUTXMTBZYXQ, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:24.694151741 (91h41m24.565234786s) [noise=true , address=145.79.227.41:4087 , piece_id=CC643HDKTLAEAAOHWOSXJL75BJH62EL7QISOZVG2VYPH4S3LDKMQ, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:24.694508199 (91h41m24.565591244s) [noise=true , address=145.79.227.39:4014 , piece_id=4F3Y2MMBAZVYJW6T5ZRNLQ4ZU56GF4EQMY6BPKZTCJ7KR7EY6UXQ, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:24.694727136 (91h41m24.565810181s) [noise=true , address=45.91.116.4:4939 , piece_id=QK4MQXBPU7SZ7SUA7SMDF5IJ2UUA6Q6MA2MA4UR3DSMU6QN7OHSQ, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:24.695257175 (91h41m24.56634022s) [noise=true , address=145.79.227.26:4655 , piece_id=WSNSSYE2G6LWCKNBMS63KSEPZBQSENSZ536OHOEAHKMLAIVQA4ZA, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:24.695748691 (91h41m24.566831736s) [noise=true , address=145.79.227.27:4689 , piece_id=4PFOTYEIDSKPMQF4LFVZZB5MK4O6JXFTU2QCLC5GUZTAUFPOE45Q, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:24.696362650 (91h41m24.567445695s) [noise=true , address=145.79.227.31:4799 , piece_id=4GDZXDMIOOWAIEKGEVXHGKF2LQ4YLSCCPA4S6UPCCAVHGRUT4DCA, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:24.698217953 (91h41m24.569300988s) [noise=true , address=77.90.31.14:4661 , piece_id=LJY3BTAERCAOCCIKW2THHEBAYKOEXDYQVGJTMWHYBACZJ4XJTBBA, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:24.701427525 (91h41m24.57251057s) [noise=true , address=77.90.31.12:4592 , piece_id=THMPWJ2JV2RVPMFHR4UPYJEC2FT7AS7QYVZQI427KGYKTYGI3ZMQ, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:24.705981576 (91h41m24.577064621s) [noise=true , address=145.79.227.33:4869 , piece_id=R7N7EKAQ6PUL7IBCY7SH4GTGQ6XYU3ZB7UJUVKFX5S7NTFRSCFNA, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:24.708340077 (91h41m24.579423112s) [noise=true , address=145.79.227.27:4690 , piece_id=OBIAVWCVVSLUDUW7G7ZGIQRIVOHO5R4KR25AVMHNA4RSTT6KB23Q, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:24.710358000 (91h41m24.581441035s) [noise=true , address=145.79.227.43:4157 , piece_id=7WZ2NUCKMURKNDQCZ6FHUC2I4XFXK4W3B3DR26DK6F2G62R7YC7Q, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:24.711000353 (91h41m24.582083388s) [noise=true , address=145.79.227.29:4943 , piece_id=PVDPIUVHEL3DLK45G2NTR3INZJ2CEPKLF6KX6CZMOMKXEOFRTTBA, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:24.712392574 (91h41m24.583475609s) [noise=true , address=77.90.31.16:4734 , piece_id=BQYJNHPRU7WKEHXUXJCUX6ZDU2RSVDPAAJTOD3VEQCEGE7TVR6CQ, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:26.323507311 (91h41m26.194590346s) [upload=79956]: Next returned nil segment Dec 9 11:10:26.323527019 (91h41m26.194610064s) [upload=79956]: Waiting for error group managing segments... Dec 9 11:10:26.705924029 (91h41m26.577007074s) [noise=true , address=145.79.227.28:4727 , piece_id=2VL43MLR2PE4BPHW4IWKNWAW3PSOAX2EV32XWLQYMYBM4FZ7RT6Q, seg_pos={0 0}, upload=79956]: Done uploading piece. err: <nil> Dec 9 11:10:26.713527548 (91h41m26.584610593s) [noise=true , address=145.79.227.42:4121 , piece_id=THP22HNINRQMLRTQJKBHRCSBE5O4JMZWVNFRTJDVFCP7CKP57V7Q, seg_pos={0 0}, upload=79956]: Done uploading piece. err: <nil> Dec 9 11:10:26.715957385 (91h41m26.58704043s) [noise=true , address=145.79.227.23:4545 , piece_id=XVIQ424I6HQC4IXH2XAYRG7DFXJ7LJ44VT7ZCWVHFFUTXMTBZYXQ, seg_pos={0 0}, upload=79956]: Done uploading piece. err: <nil> Dec 9 11:10:26.721154390 (91h41m26.592237435s) [noise=true , address=145.79.227.26:4655 , piece_id=WSNSSYE2G6LWCKNBMS63KSEPZBQSENSZ536OHOEAHKMLAIVQA4ZA, seg_pos={0 0}, upload=79956]: Done uploading piece. err: <nil> Dec 9 11:10:26.726209116 (91h41m26.597292161s) [noise=true , address=145.79.227.43:4157 , piece_id=7WZ2NUCKMURKNDQCZ6FHUC2I4XFXK4W3B3DR26DK6F2G62R7YC7Q, seg_pos={0 0}, upload=79956]: Done uploading piece. err: <nil> Dec 9 11:10:26.744717479 (91h41m26.615800524s) [noise=true , address=145.79.227.31:4799 , piece_id=4GDZXDMIOOWAIEKGEVXHGKF2LQ4YLSCCPA4S6UPCCAVHGRUT4DCA, seg_pos={0 0}, upload=79956]: Done uploading piece. err: <nil> Dec 9 11:10:26.865540120 (91h41m26.736623165s) [noise=true , address=77.90.31.14:4661 , piece_id=LJY3BTAERCAOCCIKW2THHEBAYKOEXDYQVGJTMWHYBACZJ4XJTBBA, seg_pos={0 0}, upload=79956]: Done uploading piece. err: <nil> Dec 9 11:10:26.874733556 (91h41m26.745816591s) [noise=true , address=77.90.31.12:4592 , piece_id=THMPWJ2JV2RVPMFHR4UPYJEC2FT7AS7QYVZQI427KGYKTYGI3ZMQ, seg_pos={0 0}, upload=79956]: Done uploading piece. err: <nil> Dec 9 11:10:27.552332367 (91h41m27.423415412s) [noise=true , address=145.79.227.39:4014 , piece_id=4F3Y2MMBAZVYJW6T5ZRNLQ4ZU56GF4EQMY6BPKZTCJ7KR7EY6UXQ, seg_pos={0 0}, upload=79956]: Done uploading piece. err: <nil> Dec 9 11:10:27.555135495 (91h41m27.42621853s) [noise=true , address=145.79.227.27:4689 , piece_id=4PFOTYEIDSKPMQF4LFVZZB5MK4O6JXFTU2QCLC5GUZTAUFPOE45Q, seg_pos={0 0}, upload=79956]: Done uploading piece. err: <nil> Dec 9 11:10:27.837239356 (91h41m27.708322401s) [noise=true , address=145.79.227.41:4087 , piece_id=CC643HDKTLAEAAOHWOSXJL75BJH62EL7QISOZVG2VYPH4S3LDKMQ, seg_pos={0 0}, upload=79956]: Done uploading piece. err: <nil> Dec 9 11:10:27.850249129 (91h41m27.721332174s) [noise=true , address=145.79.227.29:4943 , piece_id=PVDPIUVHEL3DLK45G2NTR3INZJ2CEPKLF6KX6CZMOMKXEOFRTTBA, seg_pos={0 0}, upload=79956]: Done uploading piece. err: <nil> Dec 9 11:10:27.850441505 (91h41m27.72152455s) [noise=true , address=145.79.227.33:4869 , piece_id=R7N7EKAQ6PUL7IBCY7SH4GTGQ6XYU3ZB7UJUVKFX5S7NTFRSCFNA, seg_pos={0 0}, upload=79956]: Done uploading piece. err: <nil> Dec 9 11:10:27.850450944 (91h41m27.721533989s) [seg_pos={0 0}, upload=79956]: Segment reached optimal threshold of 13 pieces. Dec 9 11:10:27.850729945 (91h41m27.72181298s) [noise=true , address=77.90.31.16:4734 , piece_id=BQYJNHPRU7WKEHXUXJCUX6ZDU2RSVDPAAJTOD3VEQCEGE7TVR6CQ, seg_pos={0 0}, upload=79956]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: write tcp 188.166.225.170:57470->77.90.31.16:4734: use of closed network connection; write tcp 188.166.225.170:57470->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57470->77.90.31.16:4734: use of closed network connection Dec 9 11:10:27.850737028 (91h41m27.721820063s) [noise=true , address=77.90.31.16:4734 , piece_id=BQYJNHPRU7WKEHXUXJCUX6ZDU2RSVDPAAJTOD3VEQCEGE7TVR6CQ, seg_pos={0 0}, upload=79956]: Long tail cancellation due to optimal threshold Dec 9 11:10:27.850945425 (91h41m27.72202848s) [noise=true , address=45.91.116.4:4939 , piece_id=QK4MQXBPU7SZ7SUA7SMDF5IJ2UUA6Q6MA2MA4UR3DSMU6QN7OHSQ, seg_pos={0 0}, upload=79956]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: read tcp 188.166.225.170:48538->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 9 11:10:27.850974751 (91h41m27.722057816s) [noise=true , address=45.91.116.4:4939 , piece_id=QK4MQXBPU7SZ7SUA7SMDF5IJ2UUA6Q6MA2MA4UR3DSMU6QN7OHSQ, seg_pos={0 0}, upload=79956]: Long tail cancellation due to optimal threshold Dec 9 11:10:27.851168249 (91h41m27.722251294s) [noise=true , address=145.79.227.27:4690 , piece_id=OBIAVWCVVSLUDUW7G7ZGIQRIVOHO5R4KR25AVMHNA4RSTT6KB23Q, seg_pos={0 0}, upload=79956]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H): protocol: expected piece hash; context canceled; EOF Dec 9 11:10:27.851178119 (91h41m27.722261194s) [noise=true , address=145.79.227.27:4690 , piece_id=OBIAVWCVVSLUDUW7G7ZGIQRIVOHO5R4KR25AVMHNA4RSTT6KB23Q, seg_pos={0 0}, upload=79956]: Long tail cancellation due to optimal threshold Dec 9 11:10:27.851202003 (91h41m27.722285048s) [seg_pos={0 0}, upload=79956]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 9 11:10:27.851316833 (91h41m27.722399908s) [upload=79956]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 11:10:27.851325489 (91h41m27.722408534s) [upload=79956]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 11:25:29.822344337 (91h56m29.693427372s) [upload=79956]: Done uploading segments... Dec 9 11:25:29.822415463 (91h56m29.693498498s) [upload=79956]: Upload finished. err: metaclient: context deadline exceeded Duration: 13.018118057s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 18.288904196s trace: https://jaeger.dev.storj.io/trace/7098c007ca020436 FAIL (2025-12-09 11:59:47 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , address=185.167.204.205:4029 , piece_id=WMBEVUF62ZU76TQDICPDPMG4JA7374YQFXTI5XJFSN6MJZBATS7Q, seg_pos={0 0}, upload=80368]: Uploading piece... Dec 9 11:38:49.362641042 (92h9m49.233724077s) [noise=true , address=185.167.204.206:4037 , piece_id=LO332KWEY4NLBKVGEVME2SF7UIGTEVCKLOHRCC7FLIJEPP5DTKTA, seg_pos={0 0}, upload=80368]: Uploading piece... Dec 9 11:38:49.362747554 (92h9m49.233830599s) [noise=true , address=185.167.204.206:4045 , piece_id=7RI3JKTQDSVERWTVQ4NIGZMBNAC6OPYUCTPU77HX3V74NBG7XALQ, seg_pos={0 0}, upload=80368]: Uploading piece... Dec 9 11:38:49.363050582 (92h9m49.234133627s) [noise=true , address=185.167.204.206:4059 , piece_id=HLZEIABVICZTZCKXHBKIFQJ3UVRGVV5EWHWFFL5G3E7CYGJN4GHA, seg_pos={0 0}, upload=80368]: Uploading piece... Dec 9 11:38:49.363113251 (92h9m49.234196286s) [noise=true , address=185.167.204.205:4016 , piece_id=55J2Z5J7LPDPCBKXAQ7V4BQW4U7W6UAQVF63LYH3BPFHSEQE6CEA, seg_pos={0 0}, upload=80368]: Uploading piece... Dec 9 11:38:49.363383676 (92h9m49.234466711s) [noise=true , address=185.167.204.206:4048 , piece_id=XMV7K5EEISKMT5JQFUD6EXZ3EYELHBWF6Y3J276VUUP2YGXJKF3A, seg_pos={0 0}, upload=80368]: Uploading piece... Dec 9 11:38:49.363681573 (92h9m49.234764608s) [noise=true , address=185.167.204.205:4001 , piece_id=KIUIPM26OHXISSEHSZPLFHUXF4NVOHSY6TILREFVGDRDBB5IOCQQ, seg_pos={0 0}, upload=80368]: Uploading piece... Dec 9 11:38:49.364037821 (92h9m49.235120866s) [noise=true , address=81.171.14.70:4042 , piece_id=WWJDEHBRMRVXANEWVG6B6QLD2SR7M6FXH4642UZ3AFQSDGCO2HYA, seg_pos={0 0}, upload=80368]: Uploading piece... Dec 9 11:38:49.364321191 (92h9m49.235404226s) [noise=true , address=81.171.14.70:4032 , piece_id=GOAKND7Z3IDBUYT3ILUUBU7YC3UMA6PPDJLLMFTH2OKHF76GCHWA, seg_pos={0 0}, upload=80368]: Uploading piece... Dec 9 11:38:49.364596545 (92h9m49.23567959s) [noise=true , address=81.171.14.70:4039 , piece_id=EJYQEP7LA6QJQICJQQVTNIXDN6ID5PAPL54GADSICUBJ3YQQ5E7Q, seg_pos={0 0}, upload=80368]: Uploading piece... Dec 9 11:38:49.365194575 (92h9m49.23627762s) [noise=true , address=89.149.201.147:4011 , piece_id=IJZKML6ZS5AF7KRYSHG4ZDR7AZSJGUWRSPTLSYTWR6WKYXWORAYA, seg_pos={0 0}, upload=80368]: Uploading piece... Dec 9 11:38:50.188365108 (92h9m50.059448153s) [upload=80368]: Next returned nil segment Dec 9 11:38:50.188379346 (92h9m50.059462391s) [upload=80368]: Waiting for error group managing segments... Dec 9 11:38:50.573198910 (92h9m50.444281955s) [noise=true , address=185.167.204.206:4062 , piece_id=UZUYTLORRUJOMEX62Z7ABCOXWZHBZ7PV72MA4TZIWDDWZKBR22OA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:50.577545046 (92h9m50.448628081s) [noise=true , address=185.167.204.206:4059 , piece_id=HLZEIABVICZTZCKXHBKIFQJ3UVRGVV5EWHWFFL5G3E7CYGJN4GHA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:50.577840359 (92h9m50.448923394s) [noise=true , address=185.167.204.206:4045 , piece_id=7RI3JKTQDSVERWTVQ4NIGZMBNAC6OPYUCTPU77HX3V74NBG7XALQ, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:50.578430443 (92h9m50.449513478s) [noise=true , address=185.167.204.206:4048 , piece_id=XMV7K5EEISKMT5JQFUD6EXZ3EYELHBWF6Y3J276VUUP2YGXJKF3A, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:50.579912414 (92h9m50.450995459s) [noise=true , address=185.167.204.205:4016 , piece_id=55J2Z5J7LPDPCBKXAQ7V4BQW4U7W6UAQVF63LYH3BPFHSEQE6CEA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:50.580088579 (92h9m50.451171625s) [noise=true , address=185.167.204.206:4068 , piece_id=FDF7EQNPFGNEDO6CEPYGMAAJOIT4TPHY2PVYHNXH7ILC7AY2LO3A, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:50.582475435 (92h9m50.45355848s) [noise=true , address=185.167.204.206:4037 , piece_id=LO332KWEY4NLBKVGEVME2SF7UIGTEVCKLOHRCC7FLIJEPP5DTKTA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:50.586259020 (92h9m50.457342055s) [noise=true , address=185.167.204.205:4030 , piece_id=VHBCZ6QQRBZ4THX6XCFIZ6JSFMIONQFSEZ3ERGOZB6L7L5USAKMA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:50.586458941 (92h9m50.457541986s) [noise=true , address=185.167.204.206:4060 , piece_id=A3NTK6DGBBOACMUONBGII27GVS7QZWOGDEKBPZFX76RSH5HMXS5A, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:50.588436907 (92h9m50.459519942s) [noise=true , address=185.167.204.205:4009 , piece_id=5WUDBEGUENMGEMSND2VCENBNGG2BIV47ICSWN3CJYLQQ3XIVZXKQ, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:50.589577038 (92h9m50.460660083s) [noise=true , address=185.167.204.205:4029 , piece_id=WMBEVUF62ZU76TQDICPDPMG4JA7374YQFXTI5XJFSN6MJZBATS7Q, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:50.592840462 (92h9m50.463923507s) [noise=true , address=185.167.204.205:4001 , piece_id=KIUIPM26OHXISSEHSZPLFHUXF4NVOHSY6TILREFVGDRDBB5IOCQQ, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.141501915 (92h9m51.01258496s) [noise=true , address=81.171.14.70:4039 , piece_id=EJYQEP7LA6QJQICJQQVTNIXDN6ID5PAPL54GADSICUBJ3YQQ5E7Q, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.153777871 (92h9m51.024860916s) [noise=true , address=81.171.14.70:4042 , piece_id=WWJDEHBRMRVXANEWVG6B6QLD2SR7M6FXH4642UZ3AFQSDGCO2HYA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.153948696 (92h9m51.025031751s) [noise=true , address=81.171.14.70:4028 , piece_id=NSYGSKJBIHMHZZX7HK7N3UQDGKWWQARJPGLAWH2TEPGWKNYWPC4A, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.157888027 (92h9m51.028971062s) [noise=true , address=81.171.14.70:4041 , piece_id=PZNQJLGVYH3CBPX4L2URCEVLMGK4JLQAMVFX25UBONKVYSTSEEWQ, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.170799434 (92h9m51.041882489s) [noise=true , address=81.171.14.70:4045 , piece_id=TZIZNHRR66JMWMZJNUHYFPYJ2ORR3OLKZ7VOQ5MIWXQOFARHL4VA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.171007360 (92h9m51.042090405s) [noise=true , address=81.171.14.70:4035 , piece_id=5T5MX4YDMI4IUY4LDSREN2MTN3NUL4GS6KZXDMNZMKJYY3VMS47Q, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.184999573 (92h9m51.056082648s) [noise=true , address=81.171.14.70:4032 , piece_id=GOAKND7Z3IDBUYT3ILUUBU7YC3UMA6PPDJLLMFTH2OKHF76GCHWA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.196460517 (92h9m51.067543562s) [noise=true , address=81.171.14.70:4040 , piece_id=4GA5CWKM5RJBL2IMWZNRGAA2T2JTFX54AFEFI3NAV2GNTKIDUELQ, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.199311115 (92h9m51.07039416s) [noise=true , address=81.171.14.70:4037 , piece_id=G6MBAEOMB7R4MQKZDULZY4DYQ5NR2QMCKW6HWYA55VD5YMSAERPA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.237519992 (92h9m51.108603028s) [noise=true , address=81.171.14.70:4030 , piece_id=VFAGZYZMOZKEHRNPUR2D2JCRDZFUFXUOS3JTTI2UOUYTIUY74NFQ, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.733107833 (92h9m51.604190868s) [noise=true , address=89.149.201.147:4005 , piece_id=OCHDE5VQIQPUVCOFXX343C7LXUIJFD4Q62ZZPEXKRN63KARUQR5A, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.815947736 (92h9m51.687030771s) [noise=true , address=89.149.201.147:4015 , piece_id=PGXJFJBHXVMXBTIWDTAOZKAU2JKTFSGCAD5I5MWU2MD32GF3AMDQ, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.841035780 (92h9m51.712118825s) [noise=true , address=89.149.201.147:4020 , piece_id=KIYFOE4WT3CGQQ3CL3BP2A3HH2KVKKLAUUGMHYPADVKI7BVBBRTA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.864720530 (92h9m51.735803575s) [noise=true , address=89.149.201.147:4002 , piece_id=TIK5YZA7N6MYGZ6KGNXPFBEQXD3DE2UXQ5Q2JP3EZGSLJ7TAO2GA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.864741660 (92h9m51.735824705s) [seg_pos={0 0}, upload=80368]: Segment reached optimal threshold of 26 pieces. Dec 9 11:38:51.865018136 (92h9m51.736101181s) [noise=true , address=89.149.201.147:4011 , piece_id=IJZKML6ZS5AF7KRYSHG4ZDR7AZSJGUWRSPTLSYTWR6WKYXWORAYA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PcxodnQMJERGjvBgnz3iEiC9MseRhMydkCriwGpQKqi6DGFDg): protocol: expected piece hash; context canceled; EOF Dec 9 11:38:51.865029489 (92h9m51.736112534s) [noise=true , address=89.149.201.147:4011 , piece_id=IJZKML6ZS5AF7KRYSHG4ZDR7AZSJGUWRSPTLSYTWR6WKYXWORAYA, seg_pos={0 0}, upload=80368]: Long tail cancellation due to optimal threshold Dec 9 11:38:51.865308640 (92h9m51.736391685s) [noise=true , address=89.149.201.147:4007 , piece_id=FHFZMTRZEYLGK2QOXICD5VLT77MIFP3HL6Z7SP7URQEY5L3HIDTA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Q7d4vj1KoWhDr6xd9gEfFrZ1sV3Yro2AFke2f3Ke1rXsGnvew): protocol: expected piece hash; context canceled; EOF Dec 9 11:38:51.865331703 (92h9m51.736414748s) [noise=true , address=89.149.201.147:4007 , piece_id=FHFZMTRZEYLGK2QOXICD5VLT77MIFP3HL6Z7SP7URQEY5L3HIDTA, seg_pos={0 0}, upload=80368]: Long tail cancellation due to optimal threshold Dec 9 11:38:51.865403281 (92h9m51.736486326s) [seg_pos={0 0}, upload=80368]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 9 11:38:51.865634440 (92h9m51.736717485s) [upload=80368]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 11:38:51.865643307 (92h9m51.736726352s) [upload=80368]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 11:40:37.870707645 (92h11m37.741790681s) [upload=80368]: Done uploading segments... Dec 9 11:40:37.870754294 (92h11m37.741837339s) [upload=80368]: Upload finished. err: metaclient: context deadline exceeded Duration: 18.288986903s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 5.684665486s trace: https://jaeger.dev.storj.io/trace/1ad10e1389335c60 FAIL (2025-12-09 11:59:34 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.684712125s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 7.545362835s trace: https://jaeger.dev.storj.io/trace/180c8a02a447001f FAIL (2025-12-09 11:58:53 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 1m24.245559s) [upload=79956]: Starting upload Dec 9 11:10:24.374550547 (91h41m24.245633592s) [upload=79956]: Done starting upload Dec 9 11:10:24.374695774 (91h41m24.245778819s) [upload=79956]: Uploading segments... Dec 9 11:10:24.374799832 (91h41m24.245882877s) [upload=79956]: Got next segment. Inline: false Dec 9 11:10:24.374845269 (91h41m24.245928314s) [upload=79956]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 11:10:24.374891817 (91h41m24.245974862s) [upload=79956]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 11:10:24.692953929 (91h41m24.564036964s) [seg_pos={0 0}, upload=79956]: Begin upload segment... Dec 9 11:10:24.692996710 (91h41m24.564079745s) [seg_pos={0 0}, upload=79956]: Done begin upload segment. Dec 9 11:10:24.693021047 (91h41m24.564104092s) [noise=true , address=145.79.227.42:4121 , piece_id=THP22HNINRQMLRTQJKBHRCSBE5O4JMZWVNFRTJDVFCP7CKP57V7Q, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:24.693030324 (91h41m24.56411336s) [noise=true , address=145.79.227.28:4727 , piece_id=2VL43MLR2PE4BPHW4IWKNWAW3PSOAX2EV32XWLQYMYBM4FZ7RT6Q, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:24.693620949 (91h41m24.564703994s) [noise=true , address=145.79.227.23:4545 , piece_id=XVIQ424I6HQC4IXH2XAYRG7DFXJ7LJ44VT7ZCWVHFFUTXMTBZYXQ, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:24.694151741 (91h41m24.565234786s) [noise=true , address=145.79.227.41:4087 , piece_id=CC643HDKTLAEAAOHWOSXJL75BJH62EL7QISOZVG2VYPH4S3LDKMQ, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:24.694508199 (91h41m24.565591244s) [noise=true , address=145.79.227.39:4014 , piece_id=4F3Y2MMBAZVYJW6T5ZRNLQ4ZU56GF4EQMY6BPKZTCJ7KR7EY6UXQ, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:24.694727136 (91h41m24.565810181s) [noise=true , address=45.91.116.4:4939 , piece_id=QK4MQXBPU7SZ7SUA7SMDF5IJ2UUA6Q6MA2MA4UR3DSMU6QN7OHSQ, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:24.695257175 (91h41m24.56634022s) [noise=true , address=145.79.227.26:4655 , piece_id=WSNSSYE2G6LWCKNBMS63KSEPZBQSENSZ536OHOEAHKMLAIVQA4ZA, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:24.695748691 (91h41m24.566831736s) [noise=true , address=145.79.227.27:4689 , piece_id=4PFOTYEIDSKPMQF4LFVZZB5MK4O6JXFTU2QCLC5GUZTAUFPOE45Q, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:24.696362650 (91h41m24.567445695s) [noise=true , address=145.79.227.31:4799 , piece_id=4GDZXDMIOOWAIEKGEVXHGKF2LQ4YLSCCPA4S6UPCCAVHGRUT4DCA, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:24.698217953 (91h41m24.569300988s) [noise=true , address=77.90.31.14:4661 , piece_id=LJY3BTAERCAOCCIKW2THHEBAYKOEXDYQVGJTMWHYBACZJ4XJTBBA, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:24.701427525 (91h41m24.57251057s) [noise=true , address=77.90.31.12:4592 , piece_id=THMPWJ2JV2RVPMFHR4UPYJEC2FT7AS7QYVZQI427KGYKTYGI3ZMQ, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:24.705981576 (91h41m24.577064621s) [noise=true , address=145.79.227.33:4869 , piece_id=R7N7EKAQ6PUL7IBCY7SH4GTGQ6XYU3ZB7UJUVKFX5S7NTFRSCFNA, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:24.708340077 (91h41m24.579423112s) [noise=true , address=145.79.227.27:4690 , piece_id=OBIAVWCVVSLUDUW7G7ZGIQRIVOHO5R4KR25AVMHNA4RSTT6KB23Q, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:24.710358000 (91h41m24.581441035s) [noise=true , address=145.79.227.43:4157 , piece_id=7WZ2NUCKMURKNDQCZ6FHUC2I4XFXK4W3B3DR26DK6F2G62R7YC7Q, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:24.711000353 (91h41m24.582083388s) [noise=true , address=145.79.227.29:4943 , piece_id=PVDPIUVHEL3DLK45G2NTR3INZJ2CEPKLF6KX6CZMOMKXEOFRTTBA, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:24.712392574 (91h41m24.583475609s) [noise=true , address=77.90.31.16:4734 , piece_id=BQYJNHPRU7WKEHXUXJCUX6ZDU2RSVDPAAJTOD3VEQCEGE7TVR6CQ, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:26.323507311 (91h41m26.194590346s) [upload=79956]: Next returned nil segment Dec 9 11:10:26.323527019 (91h41m26.194610064s) [upload=79956]: Waiting for error group managing segments... Dec 9 11:10:26.705924029 (91h41m26.577007074s) [noise=true , address=145.79.227.28:4727 , piece_id=2VL43MLR2PE4BPHW4IWKNWAW3PSOAX2EV32XWLQYMYBM4FZ7RT6Q, seg_pos={0 0}, upload=79956]: Done uploading piece. err: <nil> Dec 9 11:10:26.713527548 (91h41m26.584610593s) [noise=true , address=145.79.227.42:4121 , piece_id=THP22HNINRQMLRTQJKBHRCSBE5O4JMZWVNFRTJDVFCP7CKP57V7Q, seg_pos={0 0}, upload=79956]: Done uploading piece. err: <nil> Dec 9 11:10:26.715957385 (91h41m26.58704043s) [noise=true , address=145.79.227.23:4545 , piece_id=XVIQ424I6HQC4IXH2XAYRG7DFXJ7LJ44VT7ZCWVHFFUTXMTBZYXQ, seg_pos={0 0}, upload=79956]: Done uploading piece. err: <nil> Dec 9 11:10:26.721154390 (91h41m26.592237435s) [noise=true , address=145.79.227.26:4655 , piece_id=WSNSSYE2G6LWCKNBMS63KSEPZBQSENSZ536OHOEAHKMLAIVQA4ZA, seg_pos={0 0}, upload=79956]: Done uploading piece. err: <nil> Dec 9 11:10:26.726209116 (91h41m26.597292161s) [noise=true , address=145.79.227.43:4157 , piece_id=7WZ2NUCKMURKNDQCZ6FHUC2I4XFXK4W3B3DR26DK6F2G62R7YC7Q, seg_pos={0 0}, upload=79956]: Done uploading piece. err: <nil> Dec 9 11:10:26.744717479 (91h41m26.615800524s) [noise=true , address=145.79.227.31:4799 , piece_id=4GDZXDMIOOWAIEKGEVXHGKF2LQ4YLSCCPA4S6UPCCAVHGRUT4DCA, seg_pos={0 0}, upload=79956]: Done uploading piece. err: <nil> Dec 9 11:10:26.865540120 (91h41m26.736623165s) [noise=true , address=77.90.31.14:4661 , piece_id=LJY3BTAERCAOCCIKW2THHEBAYKOEXDYQVGJTMWHYBACZJ4XJTBBA, seg_pos={0 0}, upload=79956]: Done uploading piece. err: <nil> Dec 9 11:10:26.874733556 (91h41m26.745816591s) [noise=true , address=77.90.31.12:4592 , piece_id=THMPWJ2JV2RVPMFHR4UPYJEC2FT7AS7QYVZQI427KGYKTYGI3ZMQ, seg_pos={0 0}, upload=79956]: Done uploading piece. err: <nil> Dec 9 11:10:27.552332367 (91h41m27.423415412s) [noise=true , address=145.79.227.39:4014 , piece_id=4F3Y2MMBAZVYJW6T5ZRNLQ4ZU56GF4EQMY6BPKZTCJ7KR7EY6UXQ, seg_pos={0 0}, upload=79956]: Done uploading piece. err: <nil> Dec 9 11:10:27.555135495 (91h41m27.42621853s) [noise=true , address=145.79.227.27:4689 , piece_id=4PFOTYEIDSKPMQF4LFVZZB5MK4O6JXFTU2QCLC5GUZTAUFPOE45Q, seg_pos={0 0}, upload=79956]: Done uploading piece. err: <nil> Dec 9 11:10:27.837239356 (91h41m27.708322401s) [noise=true , address=145.79.227.41:4087 , piece_id=CC643HDKTLAEAAOHWOSXJL75BJH62EL7QISOZVG2VYPH4S3LDKMQ, seg_pos={0 0}, upload=79956]: Done uploading piece. err: <nil> Dec 9 11:10:27.850249129 (91h41m27.721332174s) [noise=true , address=145.79.227.29:4943 , piece_id=PVDPIUVHEL3DLK45G2NTR3INZJ2CEPKLF6KX6CZMOMKXEOFRTTBA, seg_pos={0 0}, upload=79956]: Done uploading piece. err: <nil> Dec 9 11:10:27.850441505 (91h41m27.72152455s) [noise=true , address=145.79.227.33:4869 , piece_id=R7N7EKAQ6PUL7IBCY7SH4GTGQ6XYU3ZB7UJUVKFX5S7NTFRSCFNA, seg_pos={0 0}, upload=79956]: Done uploading piece. err: <nil> Dec 9 11:10:27.850450944 (91h41m27.721533989s) [seg_pos={0 0}, upload=79956]: Segment reached optimal threshold of 13 pieces. Dec 9 11:10:27.850729945 (91h41m27.72181298s) [noise=true , address=77.90.31.16:4734 , piece_id=BQYJNHPRU7WKEHXUXJCUX6ZDU2RSVDPAAJTOD3VEQCEGE7TVR6CQ, seg_pos={0 0}, upload=79956]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: write tcp 188.166.225.170:57470->77.90.31.16:4734: use of closed network connection; write tcp 188.166.225.170:57470->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57470->77.90.31.16:4734: use of closed network connection Dec 9 11:10:27.850737028 (91h41m27.721820063s) [noise=true , address=77.90.31.16:4734 , piece_id=BQYJNHPRU7WKEHXUXJCUX6ZDU2RSVDPAAJTOD3VEQCEGE7TVR6CQ, seg_pos={0 0}, upload=79956]: Long tail cancellation due to optimal threshold Dec 9 11:10:27.850945425 (91h41m27.72202848s) [noise=true , address=45.91.116.4:4939 , piece_id=QK4MQXBPU7SZ7SUA7SMDF5IJ2UUA6Q6MA2MA4UR3DSMU6QN7OHSQ, seg_pos={0 0}, upload=79956]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: read tcp 188.166.225.170:48538->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 9 11:10:27.850974751 (91h41m27.722057816s) [noise=true , address=45.91.116.4:4939 , piece_id=QK4MQXBPU7SZ7SUA7SMDF5IJ2UUA6Q6MA2MA4UR3DSMU6QN7OHSQ, seg_pos={0 0}, upload=79956]: Long tail cancellation due to optimal threshold Dec 9 11:10:27.851168249 (91h41m27.722251294s) [noise=true , address=145.79.227.27:4690 , piece_id=OBIAVWCVVSLUDUW7G7ZGIQRIVOHO5R4KR25AVMHNA4RSTT6KB23Q, seg_pos={0 0}, upload=79956]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H): protocol: expected piece hash; context canceled; EOF Dec 9 11:10:27.851178119 (91h41m27.722261194s) [noise=true , address=145.79.227.27:4690 , piece_id=OBIAVWCVVSLUDUW7G7ZGIQRIVOHO5R4KR25AVMHNA4RSTT6KB23Q, seg_pos={0 0}, upload=79956]: Long tail cancellation due to optimal threshold Dec 9 11:10:27.851202003 (91h41m27.722285048s) [seg_pos={0 0}, upload=79956]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 9 11:10:27.851316833 (91h41m27.722399908s) [upload=79956]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 11:10:27.851325489 (91h41m27.722408534s) [upload=79956]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 11:25:29.822344337 (91h56m29.693427372s) [upload=79956]: Done uploading segments... Dec 9 11:25:29.822415463 (91h56m29.693498498s) [upload=79956]: Upload finished. err: metaclient: context deadline exceeded Duration: 7.545391098s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [..X..X............X....................XX................X..] duration: 3.428178602s trace: https://jaeger.dev.storj.io/trace/36e1a86555e8f5a8 FAIL (2025-12-09 11:58:52 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.428260188s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 475.422577ms remote: "tcp:35.207.121.91:443" target: https://eu1.storj.io/login trace: https://jaeger.dev.storj.io/trace/215c1ddb7fa82bd3 FAIL (2025-12-09 11:58:49 UTC) orbiter warning: Get "https://eu1.storj.io/login": EOF Duration: 475.452284ms core satellite eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 17.982426161s trace: https://jaeger.dev.storj.io/trace/7098c007ca020436 FAIL (2025-12-09 11:58:42 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , address=185.167.204.205:4029 , piece_id=WMBEVUF62ZU76TQDICPDPMG4JA7374YQFXTI5XJFSN6MJZBATS7Q, seg_pos={0 0}, upload=80368]: Uploading piece... Dec 9 11:38:49.362641042 (92h9m49.233724077s) [noise=true , address=185.167.204.206:4037 , piece_id=LO332KWEY4NLBKVGEVME2SF7UIGTEVCKLOHRCC7FLIJEPP5DTKTA, seg_pos={0 0}, upload=80368]: Uploading piece... Dec 9 11:38:49.362747554 (92h9m49.233830599s) [noise=true , address=185.167.204.206:4045 , piece_id=7RI3JKTQDSVERWTVQ4NIGZMBNAC6OPYUCTPU77HX3V74NBG7XALQ, seg_pos={0 0}, upload=80368]: Uploading piece... Dec 9 11:38:49.363050582 (92h9m49.234133627s) [noise=true , address=185.167.204.206:4059 , piece_id=HLZEIABVICZTZCKXHBKIFQJ3UVRGVV5EWHWFFL5G3E7CYGJN4GHA, seg_pos={0 0}, upload=80368]: Uploading piece... Dec 9 11:38:49.363113251 (92h9m49.234196286s) [noise=true , address=185.167.204.205:4016 , piece_id=55J2Z5J7LPDPCBKXAQ7V4BQW4U7W6UAQVF63LYH3BPFHSEQE6CEA, seg_pos={0 0}, upload=80368]: Uploading piece... Dec 9 11:38:49.363383676 (92h9m49.234466711s) [noise=true , address=185.167.204.206:4048 , piece_id=XMV7K5EEISKMT5JQFUD6EXZ3EYELHBWF6Y3J276VUUP2YGXJKF3A, seg_pos={0 0}, upload=80368]: Uploading piece... Dec 9 11:38:49.363681573 (92h9m49.234764608s) [noise=true , address=185.167.204.205:4001 , piece_id=KIUIPM26OHXISSEHSZPLFHUXF4NVOHSY6TILREFVGDRDBB5IOCQQ, seg_pos={0 0}, upload=80368]: Uploading piece... Dec 9 11:38:49.364037821 (92h9m49.235120866s) [noise=true , address=81.171.14.70:4042 , piece_id=WWJDEHBRMRVXANEWVG6B6QLD2SR7M6FXH4642UZ3AFQSDGCO2HYA, seg_pos={0 0}, upload=80368]: Uploading piece... Dec 9 11:38:49.364321191 (92h9m49.235404226s) [noise=true , address=81.171.14.70:4032 , piece_id=GOAKND7Z3IDBUYT3ILUUBU7YC3UMA6PPDJLLMFTH2OKHF76GCHWA, seg_pos={0 0}, upload=80368]: Uploading piece... Dec 9 11:38:49.364596545 (92h9m49.23567959s) [noise=true , address=81.171.14.70:4039 , piece_id=EJYQEP7LA6QJQICJQQVTNIXDN6ID5PAPL54GADSICUBJ3YQQ5E7Q, seg_pos={0 0}, upload=80368]: Uploading piece... Dec 9 11:38:49.365194575 (92h9m49.23627762s) [noise=true , address=89.149.201.147:4011 , piece_id=IJZKML6ZS5AF7KRYSHG4ZDR7AZSJGUWRSPTLSYTWR6WKYXWORAYA, seg_pos={0 0}, upload=80368]: Uploading piece... Dec 9 11:38:50.188365108 (92h9m50.059448153s) [upload=80368]: Next returned nil segment Dec 9 11:38:50.188379346 (92h9m50.059462391s) [upload=80368]: Waiting for error group managing segments... Dec 9 11:38:50.573198910 (92h9m50.444281955s) [noise=true , address=185.167.204.206:4062 , piece_id=UZUYTLORRUJOMEX62Z7ABCOXWZHBZ7PV72MA4TZIWDDWZKBR22OA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:50.577545046 (92h9m50.448628081s) [noise=true , address=185.167.204.206:4059 , piece_id=HLZEIABVICZTZCKXHBKIFQJ3UVRGVV5EWHWFFL5G3E7CYGJN4GHA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:50.577840359 (92h9m50.448923394s) [noise=true , address=185.167.204.206:4045 , piece_id=7RI3JKTQDSVERWTVQ4NIGZMBNAC6OPYUCTPU77HX3V74NBG7XALQ, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:50.578430443 (92h9m50.449513478s) [noise=true , address=185.167.204.206:4048 , piece_id=XMV7K5EEISKMT5JQFUD6EXZ3EYELHBWF6Y3J276VUUP2YGXJKF3A, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:50.579912414 (92h9m50.450995459s) [noise=true , address=185.167.204.205:4016 , piece_id=55J2Z5J7LPDPCBKXAQ7V4BQW4U7W6UAQVF63LYH3BPFHSEQE6CEA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:50.580088579 (92h9m50.451171625s) [noise=true , address=185.167.204.206:4068 , piece_id=FDF7EQNPFGNEDO6CEPYGMAAJOIT4TPHY2PVYHNXH7ILC7AY2LO3A, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:50.582475435 (92h9m50.45355848s) [noise=true , address=185.167.204.206:4037 , piece_id=LO332KWEY4NLBKVGEVME2SF7UIGTEVCKLOHRCC7FLIJEPP5DTKTA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:50.586259020 (92h9m50.457342055s) [noise=true , address=185.167.204.205:4030 , piece_id=VHBCZ6QQRBZ4THX6XCFIZ6JSFMIONQFSEZ3ERGOZB6L7L5USAKMA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:50.586458941 (92h9m50.457541986s) [noise=true , address=185.167.204.206:4060 , piece_id=A3NTK6DGBBOACMUONBGII27GVS7QZWOGDEKBPZFX76RSH5HMXS5A, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:50.588436907 (92h9m50.459519942s) [noise=true , address=185.167.204.205:4009 , piece_id=5WUDBEGUENMGEMSND2VCENBNGG2BIV47ICSWN3CJYLQQ3XIVZXKQ, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:50.589577038 (92h9m50.460660083s) [noise=true , address=185.167.204.205:4029 , piece_id=WMBEVUF62ZU76TQDICPDPMG4JA7374YQFXTI5XJFSN6MJZBATS7Q, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:50.592840462 (92h9m50.463923507s) [noise=true , address=185.167.204.205:4001 , piece_id=KIUIPM26OHXISSEHSZPLFHUXF4NVOHSY6TILREFVGDRDBB5IOCQQ, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.141501915 (92h9m51.01258496s) [noise=true , address=81.171.14.70:4039 , piece_id=EJYQEP7LA6QJQICJQQVTNIXDN6ID5PAPL54GADSICUBJ3YQQ5E7Q, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.153777871 (92h9m51.024860916s) [noise=true , address=81.171.14.70:4042 , piece_id=WWJDEHBRMRVXANEWVG6B6QLD2SR7M6FXH4642UZ3AFQSDGCO2HYA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.153948696 (92h9m51.025031751s) [noise=true , address=81.171.14.70:4028 , piece_id=NSYGSKJBIHMHZZX7HK7N3UQDGKWWQARJPGLAWH2TEPGWKNYWPC4A, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.157888027 (92h9m51.028971062s) [noise=true , address=81.171.14.70:4041 , piece_id=PZNQJLGVYH3CBPX4L2URCEVLMGK4JLQAMVFX25UBONKVYSTSEEWQ, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.170799434 (92h9m51.041882489s) [noise=true , address=81.171.14.70:4045 , piece_id=TZIZNHRR66JMWMZJNUHYFPYJ2ORR3OLKZ7VOQ5MIWXQOFARHL4VA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.171007360 (92h9m51.042090405s) [noise=true , address=81.171.14.70:4035 , piece_id=5T5MX4YDMI4IUY4LDSREN2MTN3NUL4GS6KZXDMNZMKJYY3VMS47Q, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.184999573 (92h9m51.056082648s) [noise=true , address=81.171.14.70:4032 , piece_id=GOAKND7Z3IDBUYT3ILUUBU7YC3UMA6PPDJLLMFTH2OKHF76GCHWA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.196460517 (92h9m51.067543562s) [noise=true , address=81.171.14.70:4040 , piece_id=4GA5CWKM5RJBL2IMWZNRGAA2T2JTFX54AFEFI3NAV2GNTKIDUELQ, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.199311115 (92h9m51.07039416s) [noise=true , address=81.171.14.70:4037 , piece_id=G6MBAEOMB7R4MQKZDULZY4DYQ5NR2QMCKW6HWYA55VD5YMSAERPA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.237519992 (92h9m51.108603028s) [noise=true , address=81.171.14.70:4030 , piece_id=VFAGZYZMOZKEHRNPUR2D2JCRDZFUFXUOS3JTTI2UOUYTIUY74NFQ, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.733107833 (92h9m51.604190868s) [noise=true , address=89.149.201.147:4005 , piece_id=OCHDE5VQIQPUVCOFXX343C7LXUIJFD4Q62ZZPEXKRN63KARUQR5A, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.815947736 (92h9m51.687030771s) [noise=true , address=89.149.201.147:4015 , piece_id=PGXJFJBHXVMXBTIWDTAOZKAU2JKTFSGCAD5I5MWU2MD32GF3AMDQ, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.841035780 (92h9m51.712118825s) [noise=true , address=89.149.201.147:4020 , piece_id=KIYFOE4WT3CGQQ3CL3BP2A3HH2KVKKLAUUGMHYPADVKI7BVBBRTA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.864720530 (92h9m51.735803575s) [noise=true , address=89.149.201.147:4002 , piece_id=TIK5YZA7N6MYGZ6KGNXPFBEQXD3DE2UXQ5Q2JP3EZGSLJ7TAO2GA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.864741660 (92h9m51.735824705s) [seg_pos={0 0}, upload=80368]: Segment reached optimal threshold of 26 pieces. Dec 9 11:38:51.865018136 (92h9m51.736101181s) [noise=true , address=89.149.201.147:4011 , piece_id=IJZKML6ZS5AF7KRYSHG4ZDR7AZSJGUWRSPTLSYTWR6WKYXWORAYA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PcxodnQMJERGjvBgnz3iEiC9MseRhMydkCriwGpQKqi6DGFDg): protocol: expected piece hash; context canceled; EOF Dec 9 11:38:51.865029489 (92h9m51.736112534s) [noise=true , address=89.149.201.147:4011 , piece_id=IJZKML6ZS5AF7KRYSHG4ZDR7AZSJGUWRSPTLSYTWR6WKYXWORAYA, seg_pos={0 0}, upload=80368]: Long tail cancellation due to optimal threshold Dec 9 11:38:51.865308640 (92h9m51.736391685s) [noise=true , address=89.149.201.147:4007 , piece_id=FHFZMTRZEYLGK2QOXICD5VLT77MIFP3HL6Z7SP7URQEY5L3HIDTA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Q7d4vj1KoWhDr6xd9gEfFrZ1sV3Yro2AFke2f3Ke1rXsGnvew): protocol: expected piece hash; context canceled; EOF Dec 9 11:38:51.865331703 (92h9m51.736414748s) [noise=true , address=89.149.201.147:4007 , piece_id=FHFZMTRZEYLGK2QOXICD5VLT77MIFP3HL6Z7SP7URQEY5L3HIDTA, seg_pos={0 0}, upload=80368]: Long tail cancellation due to optimal threshold Dec 9 11:38:51.865403281 (92h9m51.736486326s) [seg_pos={0 0}, upload=80368]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 9 11:38:51.865634440 (92h9m51.736717485s) [upload=80368]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 11:38:51.865643307 (92h9m51.736726352s) [upload=80368]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 11:40:37.870707645 (92h11m37.741790681s) [upload=80368]: Done uploading segments... Dec 9 11:40:37.870754294 (92h11m37.741837339s) [upload=80368]: Upload finished. err: metaclient: context deadline exceeded Duration: 17.982456689s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 5.697593091s trace: https://jaeger.dev.storj.io/trace/1ad10e1389335c60 FAIL (2025-12-09 11:58:40 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.697632044s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 5.984065152s trace: https://jaeger.dev.storj.io/trace/1ad10e1389335c60 FAIL (2025-12-09 11:58:23 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.984097904s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 5.777004176s trace: https://jaeger.dev.storj.io/trace/1ad10e1389335c60 FAIL (2025-12-09 11:58:00 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.777211731s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 15.574601367s trace: https://jaeger.dev.storj.io/trace/180c8a02a447001f FAIL (2025-12-09 11:57:50 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 1m24.245559s) [upload=79956]: Starting upload Dec 9 11:10:24.374550547 (91h41m24.245633592s) [upload=79956]: Done starting upload Dec 9 11:10:24.374695774 (91h41m24.245778819s) [upload=79956]: Uploading segments... Dec 9 11:10:24.374799832 (91h41m24.245882877s) [upload=79956]: Got next segment. Inline: false Dec 9 11:10:24.374845269 (91h41m24.245928314s) [upload=79956]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 11:10:24.374891817 (91h41m24.245974862s) [upload=79956]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 11:10:24.692953929 (91h41m24.564036964s) [seg_pos={0 0}, upload=79956]: Begin upload segment... Dec 9 11:10:24.692996710 (91h41m24.564079745s) [seg_pos={0 0}, upload=79956]: Done begin upload segment. Dec 9 11:10:24.693021047 (91h41m24.564104092s) [noise=true , address=145.79.227.42:4121 , piece_id=THP22HNINRQMLRTQJKBHRCSBE5O4JMZWVNFRTJDVFCP7CKP57V7Q, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:24.693030324 (91h41m24.56411336s) [noise=true , address=145.79.227.28:4727 , piece_id=2VL43MLR2PE4BPHW4IWKNWAW3PSOAX2EV32XWLQYMYBM4FZ7RT6Q, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:24.693620949 (91h41m24.564703994s) [noise=true , address=145.79.227.23:4545 , piece_id=XVIQ424I6HQC4IXH2XAYRG7DFXJ7LJ44VT7ZCWVHFFUTXMTBZYXQ, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:24.694151741 (91h41m24.565234786s) [noise=true , address=145.79.227.41:4087 , piece_id=CC643HDKTLAEAAOHWOSXJL75BJH62EL7QISOZVG2VYPH4S3LDKMQ, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:24.694508199 (91h41m24.565591244s) [noise=true , address=145.79.227.39:4014 , piece_id=4F3Y2MMBAZVYJW6T5ZRNLQ4ZU56GF4EQMY6BPKZTCJ7KR7EY6UXQ, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:24.694727136 (91h41m24.565810181s) [noise=true , address=45.91.116.4:4939 , piece_id=QK4MQXBPU7SZ7SUA7SMDF5IJ2UUA6Q6MA2MA4UR3DSMU6QN7OHSQ, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:24.695257175 (91h41m24.56634022s) [noise=true , address=145.79.227.26:4655 , piece_id=WSNSSYE2G6LWCKNBMS63KSEPZBQSENSZ536OHOEAHKMLAIVQA4ZA, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:24.695748691 (91h41m24.566831736s) [noise=true , address=145.79.227.27:4689 , piece_id=4PFOTYEIDSKPMQF4LFVZZB5MK4O6JXFTU2QCLC5GUZTAUFPOE45Q, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:24.696362650 (91h41m24.567445695s) [noise=true , address=145.79.227.31:4799 , piece_id=4GDZXDMIOOWAIEKGEVXHGKF2LQ4YLSCCPA4S6UPCCAVHGRUT4DCA, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:24.698217953 (91h41m24.569300988s) [noise=true , address=77.90.31.14:4661 , piece_id=LJY3BTAERCAOCCIKW2THHEBAYKOEXDYQVGJTMWHYBACZJ4XJTBBA, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:24.701427525 (91h41m24.57251057s) [noise=true , address=77.90.31.12:4592 , piece_id=THMPWJ2JV2RVPMFHR4UPYJEC2FT7AS7QYVZQI427KGYKTYGI3ZMQ, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:24.705981576 (91h41m24.577064621s) [noise=true , address=145.79.227.33:4869 , piece_id=R7N7EKAQ6PUL7IBCY7SH4GTGQ6XYU3ZB7UJUVKFX5S7NTFRSCFNA, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:24.708340077 (91h41m24.579423112s) [noise=true , address=145.79.227.27:4690 , piece_id=OBIAVWCVVSLUDUW7G7ZGIQRIVOHO5R4KR25AVMHNA4RSTT6KB23Q, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:24.710358000 (91h41m24.581441035s) [noise=true , address=145.79.227.43:4157 , piece_id=7WZ2NUCKMURKNDQCZ6FHUC2I4XFXK4W3B3DR26DK6F2G62R7YC7Q, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:24.711000353 (91h41m24.582083388s) [noise=true , address=145.79.227.29:4943 , piece_id=PVDPIUVHEL3DLK45G2NTR3INZJ2CEPKLF6KX6CZMOMKXEOFRTTBA, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:24.712392574 (91h41m24.583475609s) [noise=true , address=77.90.31.16:4734 , piece_id=BQYJNHPRU7WKEHXUXJCUX6ZDU2RSVDPAAJTOD3VEQCEGE7TVR6CQ, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:26.323507311 (91h41m26.194590346s) [upload=79956]: Next returned nil segment Dec 9 11:10:26.323527019 (91h41m26.194610064s) [upload=79956]: Waiting for error group managing segments... Dec 9 11:10:26.705924029 (91h41m26.577007074s) [noise=true , address=145.79.227.28:4727 , piece_id=2VL43MLR2PE4BPHW4IWKNWAW3PSOAX2EV32XWLQYMYBM4FZ7RT6Q, seg_pos={0 0}, upload=79956]: Done uploading piece. err: <nil> Dec 9 11:10:26.713527548 (91h41m26.584610593s) [noise=true , address=145.79.227.42:4121 , piece_id=THP22HNINRQMLRTQJKBHRCSBE5O4JMZWVNFRTJDVFCP7CKP57V7Q, seg_pos={0 0}, upload=79956]: Done uploading piece. err: <nil> Dec 9 11:10:26.715957385 (91h41m26.58704043s) [noise=true , address=145.79.227.23:4545 , piece_id=XVIQ424I6HQC4IXH2XAYRG7DFXJ7LJ44VT7ZCWVHFFUTXMTBZYXQ, seg_pos={0 0}, upload=79956]: Done uploading piece. err: <nil> Dec 9 11:10:26.721154390 (91h41m26.592237435s) [noise=true , address=145.79.227.26:4655 , piece_id=WSNSSYE2G6LWCKNBMS63KSEPZBQSENSZ536OHOEAHKMLAIVQA4ZA, seg_pos={0 0}, upload=79956]: Done uploading piece. err: <nil> Dec 9 11:10:26.726209116 (91h41m26.597292161s) [noise=true , address=145.79.227.43:4157 , piece_id=7WZ2NUCKMURKNDQCZ6FHUC2I4XFXK4W3B3DR26DK6F2G62R7YC7Q, seg_pos={0 0}, upload=79956]: Done uploading piece. err: <nil> Dec 9 11:10:26.744717479 (91h41m26.615800524s) [noise=true , address=145.79.227.31:4799 , piece_id=4GDZXDMIOOWAIEKGEVXHGKF2LQ4YLSCCPA4S6UPCCAVHGRUT4DCA, seg_pos={0 0}, upload=79956]: Done uploading piece. err: <nil> Dec 9 11:10:26.865540120 (91h41m26.736623165s) [noise=true , address=77.90.31.14:4661 , piece_id=LJY3BTAERCAOCCIKW2THHEBAYKOEXDYQVGJTMWHYBACZJ4XJTBBA, seg_pos={0 0}, upload=79956]: Done uploading piece. err: <nil> Dec 9 11:10:26.874733556 (91h41m26.745816591s) [noise=true , address=77.90.31.12:4592 , piece_id=THMPWJ2JV2RVPMFHR4UPYJEC2FT7AS7QYVZQI427KGYKTYGI3ZMQ, seg_pos={0 0}, upload=79956]: Done uploading piece. err: <nil> Dec 9 11:10:27.552332367 (91h41m27.423415412s) [noise=true , address=145.79.227.39:4014 , piece_id=4F3Y2MMBAZVYJW6T5ZRNLQ4ZU56GF4EQMY6BPKZTCJ7KR7EY6UXQ, seg_pos={0 0}, upload=79956]: Done uploading piece. err: <nil> Dec 9 11:10:27.555135495 (91h41m27.42621853s) [noise=true , address=145.79.227.27:4689 , piece_id=4PFOTYEIDSKPMQF4LFVZZB5MK4O6JXFTU2QCLC5GUZTAUFPOE45Q, seg_pos={0 0}, upload=79956]: Done uploading piece. err: <nil> Dec 9 11:10:27.837239356 (91h41m27.708322401s) [noise=true , address=145.79.227.41:4087 , piece_id=CC643HDKTLAEAAOHWOSXJL75BJH62EL7QISOZVG2VYPH4S3LDKMQ, seg_pos={0 0}, upload=79956]: Done uploading piece. err: <nil> Dec 9 11:10:27.850249129 (91h41m27.721332174s) [noise=true , address=145.79.227.29:4943 , piece_id=PVDPIUVHEL3DLK45G2NTR3INZJ2CEPKLF6KX6CZMOMKXEOFRTTBA, seg_pos={0 0}, upload=79956]: Done uploading piece. err: <nil> Dec 9 11:10:27.850441505 (91h41m27.72152455s) [noise=true , address=145.79.227.33:4869 , piece_id=R7N7EKAQ6PUL7IBCY7SH4GTGQ6XYU3ZB7UJUVKFX5S7NTFRSCFNA, seg_pos={0 0}, upload=79956]: Done uploading piece. err: <nil> Dec 9 11:10:27.850450944 (91h41m27.721533989s) [seg_pos={0 0}, upload=79956]: Segment reached optimal threshold of 13 pieces. Dec 9 11:10:27.850729945 (91h41m27.72181298s) [noise=true , address=77.90.31.16:4734 , piece_id=BQYJNHPRU7WKEHXUXJCUX6ZDU2RSVDPAAJTOD3VEQCEGE7TVR6CQ, seg_pos={0 0}, upload=79956]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: write tcp 188.166.225.170:57470->77.90.31.16:4734: use of closed network connection; write tcp 188.166.225.170:57470->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57470->77.90.31.16:4734: use of closed network connection Dec 9 11:10:27.850737028 (91h41m27.721820063s) [noise=true , address=77.90.31.16:4734 , piece_id=BQYJNHPRU7WKEHXUXJCUX6ZDU2RSVDPAAJTOD3VEQCEGE7TVR6CQ, seg_pos={0 0}, upload=79956]: Long tail cancellation due to optimal threshold Dec 9 11:10:27.850945425 (91h41m27.72202848s) [noise=true , address=45.91.116.4:4939 , piece_id=QK4MQXBPU7SZ7SUA7SMDF5IJ2UUA6Q6MA2MA4UR3DSMU6QN7OHSQ, seg_pos={0 0}, upload=79956]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: read tcp 188.166.225.170:48538->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 9 11:10:27.850974751 (91h41m27.722057816s) [noise=true , address=45.91.116.4:4939 , piece_id=QK4MQXBPU7SZ7SUA7SMDF5IJ2UUA6Q6MA2MA4UR3DSMU6QN7OHSQ, seg_pos={0 0}, upload=79956]: Long tail cancellation due to optimal threshold Dec 9 11:10:27.851168249 (91h41m27.722251294s) [noise=true , address=145.79.227.27:4690 , piece_id=OBIAVWCVVSLUDUW7G7ZGIQRIVOHO5R4KR25AVMHNA4RSTT6KB23Q, seg_pos={0 0}, upload=79956]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H): protocol: expected piece hash; context canceled; EOF Dec 9 11:10:27.851178119 (91h41m27.722261194s) [noise=true , address=145.79.227.27:4690 , piece_id=OBIAVWCVVSLUDUW7G7ZGIQRIVOHO5R4KR25AVMHNA4RSTT6KB23Q, seg_pos={0 0}, upload=79956]: Long tail cancellation due to optimal threshold Dec 9 11:10:27.851202003 (91h41m27.722285048s) [seg_pos={0 0}, upload=79956]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 9 11:10:27.851316833 (91h41m27.722399908s) [upload=79956]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 11:10:27.851325489 (91h41m27.722408534s) [upload=79956]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 11:25:29.822344337 (91h56m29.693427372s) [upload=79956]: Done uploading segments... Dec 9 11:25:29.822415463 (91h56m29.693498498s) [upload=79956]: Upload finished. err: metaclient: context deadline exceeded Duration: 15.574672602s core uplink us-select-1
checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 470.490434ms remote: "tcp:35.207.121.91:443" target: https://eu1.storj.io/login trace: https://jaeger.dev.storj.io/trace/215c1ddb7fa82bd3 FAIL (2025-12-09 11:57:46 UTC) orbiter warning: Get "https://eu1.storj.io/login": EOF Duration: 470.517186ms core satellite eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [...X..X............X....................XX................X.] duration: 3.563105048s trace: https://jaeger.dev.storj.io/trace/36e1a86555e8f5a8 FAIL (2025-12-09 11:57:46 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.563120909s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 5.225027174s trace: https://jaeger.dev.storj.io/trace/7098c007ca020436 FAIL (2025-12-09 11:57:40 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , address=185.167.204.205:4029 , piece_id=WMBEVUF62ZU76TQDICPDPMG4JA7374YQFXTI5XJFSN6MJZBATS7Q, seg_pos={0 0}, upload=80368]: Uploading piece... Dec 9 11:38:49.362641042 (92h9m49.233724077s) [noise=true , address=185.167.204.206:4037 , piece_id=LO332KWEY4NLBKVGEVME2SF7UIGTEVCKLOHRCC7FLIJEPP5DTKTA, seg_pos={0 0}, upload=80368]: Uploading piece... Dec 9 11:38:49.362747554 (92h9m49.233830599s) [noise=true , address=185.167.204.206:4045 , piece_id=7RI3JKTQDSVERWTVQ4NIGZMBNAC6OPYUCTPU77HX3V74NBG7XALQ, seg_pos={0 0}, upload=80368]: Uploading piece... Dec 9 11:38:49.363050582 (92h9m49.234133627s) [noise=true , address=185.167.204.206:4059 , piece_id=HLZEIABVICZTZCKXHBKIFQJ3UVRGVV5EWHWFFL5G3E7CYGJN4GHA, seg_pos={0 0}, upload=80368]: Uploading piece... Dec 9 11:38:49.363113251 (92h9m49.234196286s) [noise=true , address=185.167.204.205:4016 , piece_id=55J2Z5J7LPDPCBKXAQ7V4BQW4U7W6UAQVF63LYH3BPFHSEQE6CEA, seg_pos={0 0}, upload=80368]: Uploading piece... Dec 9 11:38:49.363383676 (92h9m49.234466711s) [noise=true , address=185.167.204.206:4048 , piece_id=XMV7K5EEISKMT5JQFUD6EXZ3EYELHBWF6Y3J276VUUP2YGXJKF3A, seg_pos={0 0}, upload=80368]: Uploading piece... Dec 9 11:38:49.363681573 (92h9m49.234764608s) [noise=true , address=185.167.204.205:4001 , piece_id=KIUIPM26OHXISSEHSZPLFHUXF4NVOHSY6TILREFVGDRDBB5IOCQQ, seg_pos={0 0}, upload=80368]: Uploading piece... Dec 9 11:38:49.364037821 (92h9m49.235120866s) [noise=true , address=81.171.14.70:4042 , piece_id=WWJDEHBRMRVXANEWVG6B6QLD2SR7M6FXH4642UZ3AFQSDGCO2HYA, seg_pos={0 0}, upload=80368]: Uploading piece... Dec 9 11:38:49.364321191 (92h9m49.235404226s) [noise=true , address=81.171.14.70:4032 , piece_id=GOAKND7Z3IDBUYT3ILUUBU7YC3UMA6PPDJLLMFTH2OKHF76GCHWA, seg_pos={0 0}, upload=80368]: Uploading piece... Dec 9 11:38:49.364596545 (92h9m49.23567959s) [noise=true , address=81.171.14.70:4039 , piece_id=EJYQEP7LA6QJQICJQQVTNIXDN6ID5PAPL54GADSICUBJ3YQQ5E7Q, seg_pos={0 0}, upload=80368]: Uploading piece... Dec 9 11:38:49.365194575 (92h9m49.23627762s) [noise=true , address=89.149.201.147:4011 , piece_id=IJZKML6ZS5AF7KRYSHG4ZDR7AZSJGUWRSPTLSYTWR6WKYXWORAYA, seg_pos={0 0}, upload=80368]: Uploading piece... Dec 9 11:38:50.188365108 (92h9m50.059448153s) [upload=80368]: Next returned nil segment Dec 9 11:38:50.188379346 (92h9m50.059462391s) [upload=80368]: Waiting for error group managing segments... Dec 9 11:38:50.573198910 (92h9m50.444281955s) [noise=true , address=185.167.204.206:4062 , piece_id=UZUYTLORRUJOMEX62Z7ABCOXWZHBZ7PV72MA4TZIWDDWZKBR22OA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:50.577545046 (92h9m50.448628081s) [noise=true , address=185.167.204.206:4059 , piece_id=HLZEIABVICZTZCKXHBKIFQJ3UVRGVV5EWHWFFL5G3E7CYGJN4GHA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:50.577840359 (92h9m50.448923394s) [noise=true , address=185.167.204.206:4045 , piece_id=7RI3JKTQDSVERWTVQ4NIGZMBNAC6OPYUCTPU77HX3V74NBG7XALQ, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:50.578430443 (92h9m50.449513478s) [noise=true , address=185.167.204.206:4048 , piece_id=XMV7K5EEISKMT5JQFUD6EXZ3EYELHBWF6Y3J276VUUP2YGXJKF3A, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:50.579912414 (92h9m50.450995459s) [noise=true , address=185.167.204.205:4016 , piece_id=55J2Z5J7LPDPCBKXAQ7V4BQW4U7W6UAQVF63LYH3BPFHSEQE6CEA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:50.580088579 (92h9m50.451171625s) [noise=true , address=185.167.204.206:4068 , piece_id=FDF7EQNPFGNEDO6CEPYGMAAJOIT4TPHY2PVYHNXH7ILC7AY2LO3A, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:50.582475435 (92h9m50.45355848s) [noise=true , address=185.167.204.206:4037 , piece_id=LO332KWEY4NLBKVGEVME2SF7UIGTEVCKLOHRCC7FLIJEPP5DTKTA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:50.586259020 (92h9m50.457342055s) [noise=true , address=185.167.204.205:4030 , piece_id=VHBCZ6QQRBZ4THX6XCFIZ6JSFMIONQFSEZ3ERGOZB6L7L5USAKMA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:50.586458941 (92h9m50.457541986s) [noise=true , address=185.167.204.206:4060 , piece_id=A3NTK6DGBBOACMUONBGII27GVS7QZWOGDEKBPZFX76RSH5HMXS5A, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:50.588436907 (92h9m50.459519942s) [noise=true , address=185.167.204.205:4009 , piece_id=5WUDBEGUENMGEMSND2VCENBNGG2BIV47ICSWN3CJYLQQ3XIVZXKQ, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:50.589577038 (92h9m50.460660083s) [noise=true , address=185.167.204.205:4029 , piece_id=WMBEVUF62ZU76TQDICPDPMG4JA7374YQFXTI5XJFSN6MJZBATS7Q, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:50.592840462 (92h9m50.463923507s) [noise=true , address=185.167.204.205:4001 , piece_id=KIUIPM26OHXISSEHSZPLFHUXF4NVOHSY6TILREFVGDRDBB5IOCQQ, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.141501915 (92h9m51.01258496s) [noise=true , address=81.171.14.70:4039 , piece_id=EJYQEP7LA6QJQICJQQVTNIXDN6ID5PAPL54GADSICUBJ3YQQ5E7Q, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.153777871 (92h9m51.024860916s) [noise=true , address=81.171.14.70:4042 , piece_id=WWJDEHBRMRVXANEWVG6B6QLD2SR7M6FXH4642UZ3AFQSDGCO2HYA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.153948696 (92h9m51.025031751s) [noise=true , address=81.171.14.70:4028 , piece_id=NSYGSKJBIHMHZZX7HK7N3UQDGKWWQARJPGLAWH2TEPGWKNYWPC4A, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.157888027 (92h9m51.028971062s) [noise=true , address=81.171.14.70:4041 , piece_id=PZNQJLGVYH3CBPX4L2URCEVLMGK4JLQAMVFX25UBONKVYSTSEEWQ, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.170799434 (92h9m51.041882489s) [noise=true , address=81.171.14.70:4045 , piece_id=TZIZNHRR66JMWMZJNUHYFPYJ2ORR3OLKZ7VOQ5MIWXQOFARHL4VA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.171007360 (92h9m51.042090405s) [noise=true , address=81.171.14.70:4035 , piece_id=5T5MX4YDMI4IUY4LDSREN2MTN3NUL4GS6KZXDMNZMKJYY3VMS47Q, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.184999573 (92h9m51.056082648s) [noise=true , address=81.171.14.70:4032 , piece_id=GOAKND7Z3IDBUYT3ILUUBU7YC3UMA6PPDJLLMFTH2OKHF76GCHWA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.196460517 (92h9m51.067543562s) [noise=true , address=81.171.14.70:4040 , piece_id=4GA5CWKM5RJBL2IMWZNRGAA2T2JTFX54AFEFI3NAV2GNTKIDUELQ, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.199311115 (92h9m51.07039416s) [noise=true , address=81.171.14.70:4037 , piece_id=G6MBAEOMB7R4MQKZDULZY4DYQ5NR2QMCKW6HWYA55VD5YMSAERPA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.237519992 (92h9m51.108603028s) [noise=true , address=81.171.14.70:4030 , piece_id=VFAGZYZMOZKEHRNPUR2D2JCRDZFUFXUOS3JTTI2UOUYTIUY74NFQ, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.733107833 (92h9m51.604190868s) [noise=true , address=89.149.201.147:4005 , piece_id=OCHDE5VQIQPUVCOFXX343C7LXUIJFD4Q62ZZPEXKRN63KARUQR5A, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.815947736 (92h9m51.687030771s) [noise=true , address=89.149.201.147:4015 , piece_id=PGXJFJBHXVMXBTIWDTAOZKAU2JKTFSGCAD5I5MWU2MD32GF3AMDQ, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.841035780 (92h9m51.712118825s) [noise=true , address=89.149.201.147:4020 , piece_id=KIYFOE4WT3CGQQ3CL3BP2A3HH2KVKKLAUUGMHYPADVKI7BVBBRTA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.864720530 (92h9m51.735803575s) [noise=true , address=89.149.201.147:4002 , piece_id=TIK5YZA7N6MYGZ6KGNXPFBEQXD3DE2UXQ5Q2JP3EZGSLJ7TAO2GA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.864741660 (92h9m51.735824705s) [seg_pos={0 0}, upload=80368]: Segment reached optimal threshold of 26 pieces. Dec 9 11:38:51.865018136 (92h9m51.736101181s) [noise=true , address=89.149.201.147:4011 , piece_id=IJZKML6ZS5AF7KRYSHG4ZDR7AZSJGUWRSPTLSYTWR6WKYXWORAYA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PcxodnQMJERGjvBgnz3iEiC9MseRhMydkCriwGpQKqi6DGFDg): protocol: expected piece hash; context canceled; EOF Dec 9 11:38:51.865029489 (92h9m51.736112534s) [noise=true , address=89.149.201.147:4011 , piece_id=IJZKML6ZS5AF7KRYSHG4ZDR7AZSJGUWRSPTLSYTWR6WKYXWORAYA, seg_pos={0 0}, upload=80368]: Long tail cancellation due to optimal threshold Dec 9 11:38:51.865308640 (92h9m51.736391685s) [noise=true , address=89.149.201.147:4007 , piece_id=FHFZMTRZEYLGK2QOXICD5VLT77MIFP3HL6Z7SP7URQEY5L3HIDTA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Q7d4vj1KoWhDr6xd9gEfFrZ1sV3Yro2AFke2f3Ke1rXsGnvew): protocol: expected piece hash; context canceled; EOF Dec 9 11:38:51.865331703 (92h9m51.736414748s) [noise=true , address=89.149.201.147:4007 , piece_id=FHFZMTRZEYLGK2QOXICD5VLT77MIFP3HL6Z7SP7URQEY5L3HIDTA, seg_pos={0 0}, upload=80368]: Long tail cancellation due to optimal threshold Dec 9 11:38:51.865403281 (92h9m51.736486326s) [seg_pos={0 0}, upload=80368]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 9 11:38:51.865634440 (92h9m51.736717485s) [upload=80368]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 11:38:51.865643307 (92h9m51.736726352s) [upload=80368]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 11:40:37.870707645 (92h11m37.741790681s) [upload=80368]: Done uploading segments... Dec 9 11:40:37.870754294 (92h11m37.741837339s) [upload=80368]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.225087639s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 13.466552134s trace: https://jaeger.dev.storj.io/trace/7098c007ca020436 FAIL (2025-12-09 11:57:10 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , address=185.167.204.205:4029 , piece_id=WMBEVUF62ZU76TQDICPDPMG4JA7374YQFXTI5XJFSN6MJZBATS7Q, seg_pos={0 0}, upload=80368]: Uploading piece... Dec 9 11:38:49.362641042 (92h9m49.233724077s) [noise=true , address=185.167.204.206:4037 , piece_id=LO332KWEY4NLBKVGEVME2SF7UIGTEVCKLOHRCC7FLIJEPP5DTKTA, seg_pos={0 0}, upload=80368]: Uploading piece... Dec 9 11:38:49.362747554 (92h9m49.233830599s) [noise=true , address=185.167.204.206:4045 , piece_id=7RI3JKTQDSVERWTVQ4NIGZMBNAC6OPYUCTPU77HX3V74NBG7XALQ, seg_pos={0 0}, upload=80368]: Uploading piece... Dec 9 11:38:49.363050582 (92h9m49.234133627s) [noise=true , address=185.167.204.206:4059 , piece_id=HLZEIABVICZTZCKXHBKIFQJ3UVRGVV5EWHWFFL5G3E7CYGJN4GHA, seg_pos={0 0}, upload=80368]: Uploading piece... Dec 9 11:38:49.363113251 (92h9m49.234196286s) [noise=true , address=185.167.204.205:4016 , piece_id=55J2Z5J7LPDPCBKXAQ7V4BQW4U7W6UAQVF63LYH3BPFHSEQE6CEA, seg_pos={0 0}, upload=80368]: Uploading piece... Dec 9 11:38:49.363383676 (92h9m49.234466711s) [noise=true , address=185.167.204.206:4048 , piece_id=XMV7K5EEISKMT5JQFUD6EXZ3EYELHBWF6Y3J276VUUP2YGXJKF3A, seg_pos={0 0}, upload=80368]: Uploading piece... Dec 9 11:38:49.363681573 (92h9m49.234764608s) [noise=true , address=185.167.204.205:4001 , piece_id=KIUIPM26OHXISSEHSZPLFHUXF4NVOHSY6TILREFVGDRDBB5IOCQQ, seg_pos={0 0}, upload=80368]: Uploading piece... Dec 9 11:38:49.364037821 (92h9m49.235120866s) [noise=true , address=81.171.14.70:4042 , piece_id=WWJDEHBRMRVXANEWVG6B6QLD2SR7M6FXH4642UZ3AFQSDGCO2HYA, seg_pos={0 0}, upload=80368]: Uploading piece... Dec 9 11:38:49.364321191 (92h9m49.235404226s) [noise=true , address=81.171.14.70:4032 , piece_id=GOAKND7Z3IDBUYT3ILUUBU7YC3UMA6PPDJLLMFTH2OKHF76GCHWA, seg_pos={0 0}, upload=80368]: Uploading piece... Dec 9 11:38:49.364596545 (92h9m49.23567959s) [noise=true , address=81.171.14.70:4039 , piece_id=EJYQEP7LA6QJQICJQQVTNIXDN6ID5PAPL54GADSICUBJ3YQQ5E7Q, seg_pos={0 0}, upload=80368]: Uploading piece... Dec 9 11:38:49.365194575 (92h9m49.23627762s) [noise=true , address=89.149.201.147:4011 , piece_id=IJZKML6ZS5AF7KRYSHG4ZDR7AZSJGUWRSPTLSYTWR6WKYXWORAYA, seg_pos={0 0}, upload=80368]: Uploading piece... Dec 9 11:38:50.188365108 (92h9m50.059448153s) [upload=80368]: Next returned nil segment Dec 9 11:38:50.188379346 (92h9m50.059462391s) [upload=80368]: Waiting for error group managing segments... Dec 9 11:38:50.573198910 (92h9m50.444281955s) [noise=true , address=185.167.204.206:4062 , piece_id=UZUYTLORRUJOMEX62Z7ABCOXWZHBZ7PV72MA4TZIWDDWZKBR22OA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:50.577545046 (92h9m50.448628081s) [noise=true , address=185.167.204.206:4059 , piece_id=HLZEIABVICZTZCKXHBKIFQJ3UVRGVV5EWHWFFL5G3E7CYGJN4GHA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:50.577840359 (92h9m50.448923394s) [noise=true , address=185.167.204.206:4045 , piece_id=7RI3JKTQDSVERWTVQ4NIGZMBNAC6OPYUCTPU77HX3V74NBG7XALQ, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:50.578430443 (92h9m50.449513478s) [noise=true , address=185.167.204.206:4048 , piece_id=XMV7K5EEISKMT5JQFUD6EXZ3EYELHBWF6Y3J276VUUP2YGXJKF3A, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:50.579912414 (92h9m50.450995459s) [noise=true , address=185.167.204.205:4016 , piece_id=55J2Z5J7LPDPCBKXAQ7V4BQW4U7W6UAQVF63LYH3BPFHSEQE6CEA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:50.580088579 (92h9m50.451171625s) [noise=true , address=185.167.204.206:4068 , piece_id=FDF7EQNPFGNEDO6CEPYGMAAJOIT4TPHY2PVYHNXH7ILC7AY2LO3A, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:50.582475435 (92h9m50.45355848s) [noise=true , address=185.167.204.206:4037 , piece_id=LO332KWEY4NLBKVGEVME2SF7UIGTEVCKLOHRCC7FLIJEPP5DTKTA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:50.586259020 (92h9m50.457342055s) [noise=true , address=185.167.204.205:4030 , piece_id=VHBCZ6QQRBZ4THX6XCFIZ6JSFMIONQFSEZ3ERGOZB6L7L5USAKMA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:50.586458941 (92h9m50.457541986s) [noise=true , address=185.167.204.206:4060 , piece_id=A3NTK6DGBBOACMUONBGII27GVS7QZWOGDEKBPZFX76RSH5HMXS5A, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:50.588436907 (92h9m50.459519942s) [noise=true , address=185.167.204.205:4009 , piece_id=5WUDBEGUENMGEMSND2VCENBNGG2BIV47ICSWN3CJYLQQ3XIVZXKQ, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:50.589577038 (92h9m50.460660083s) [noise=true , address=185.167.204.205:4029 , piece_id=WMBEVUF62ZU76TQDICPDPMG4JA7374YQFXTI5XJFSN6MJZBATS7Q, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:50.592840462 (92h9m50.463923507s) [noise=true , address=185.167.204.205:4001 , piece_id=KIUIPM26OHXISSEHSZPLFHUXF4NVOHSY6TILREFVGDRDBB5IOCQQ, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.141501915 (92h9m51.01258496s) [noise=true , address=81.171.14.70:4039 , piece_id=EJYQEP7LA6QJQICJQQVTNIXDN6ID5PAPL54GADSICUBJ3YQQ5E7Q, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.153777871 (92h9m51.024860916s) [noise=true , address=81.171.14.70:4042 , piece_id=WWJDEHBRMRVXANEWVG6B6QLD2SR7M6FXH4642UZ3AFQSDGCO2HYA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.153948696 (92h9m51.025031751s) [noise=true , address=81.171.14.70:4028 , piece_id=NSYGSKJBIHMHZZX7HK7N3UQDGKWWQARJPGLAWH2TEPGWKNYWPC4A, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.157888027 (92h9m51.028971062s) [noise=true , address=81.171.14.70:4041 , piece_id=PZNQJLGVYH3CBPX4L2URCEVLMGK4JLQAMVFX25UBONKVYSTSEEWQ, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.170799434 (92h9m51.041882489s) [noise=true , address=81.171.14.70:4045 , piece_id=TZIZNHRR66JMWMZJNUHYFPYJ2ORR3OLKZ7VOQ5MIWXQOFARHL4VA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.171007360 (92h9m51.042090405s) [noise=true , address=81.171.14.70:4035 , piece_id=5T5MX4YDMI4IUY4LDSREN2MTN3NUL4GS6KZXDMNZMKJYY3VMS47Q, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.184999573 (92h9m51.056082648s) [noise=true , address=81.171.14.70:4032 , piece_id=GOAKND7Z3IDBUYT3ILUUBU7YC3UMA6PPDJLLMFTH2OKHF76GCHWA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.196460517 (92h9m51.067543562s) [noise=true , address=81.171.14.70:4040 , piece_id=4GA5CWKM5RJBL2IMWZNRGAA2T2JTFX54AFEFI3NAV2GNTKIDUELQ, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.199311115 (92h9m51.07039416s) [noise=true , address=81.171.14.70:4037 , piece_id=G6MBAEOMB7R4MQKZDULZY4DYQ5NR2QMCKW6HWYA55VD5YMSAERPA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.237519992 (92h9m51.108603028s) [noise=true , address=81.171.14.70:4030 , piece_id=VFAGZYZMOZKEHRNPUR2D2JCRDZFUFXUOS3JTTI2UOUYTIUY74NFQ, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.733107833 (92h9m51.604190868s) [noise=true , address=89.149.201.147:4005 , piece_id=OCHDE5VQIQPUVCOFXX343C7LXUIJFD4Q62ZZPEXKRN63KARUQR5A, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.815947736 (92h9m51.687030771s) [noise=true , address=89.149.201.147:4015 , piece_id=PGXJFJBHXVMXBTIWDTAOZKAU2JKTFSGCAD5I5MWU2MD32GF3AMDQ, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.841035780 (92h9m51.712118825s) [noise=true , address=89.149.201.147:4020 , piece_id=KIYFOE4WT3CGQQ3CL3BP2A3HH2KVKKLAUUGMHYPADVKI7BVBBRTA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.864720530 (92h9m51.735803575s) [noise=true , address=89.149.201.147:4002 , piece_id=TIK5YZA7N6MYGZ6KGNXPFBEQXD3DE2UXQ5Q2JP3EZGSLJ7TAO2GA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.864741660 (92h9m51.735824705s) [seg_pos={0 0}, upload=80368]: Segment reached optimal threshold of 26 pieces. Dec 9 11:38:51.865018136 (92h9m51.736101181s) [noise=true , address=89.149.201.147:4011 , piece_id=IJZKML6ZS5AF7KRYSHG4ZDR7AZSJGUWRSPTLSYTWR6WKYXWORAYA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PcxodnQMJERGjvBgnz3iEiC9MseRhMydkCriwGpQKqi6DGFDg): protocol: expected piece hash; context canceled; EOF Dec 9 11:38:51.865029489 (92h9m51.736112534s) [noise=true , address=89.149.201.147:4011 , piece_id=IJZKML6ZS5AF7KRYSHG4ZDR7AZSJGUWRSPTLSYTWR6WKYXWORAYA, seg_pos={0 0}, upload=80368]: Long tail cancellation due to optimal threshold Dec 9 11:38:51.865308640 (92h9m51.736391685s) [noise=true , address=89.149.201.147:4007 , piece_id=FHFZMTRZEYLGK2QOXICD5VLT77MIFP3HL6Z7SP7URQEY5L3HIDTA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Q7d4vj1KoWhDr6xd9gEfFrZ1sV3Yro2AFke2f3Ke1rXsGnvew): protocol: expected piece hash; context canceled; EOF Dec 9 11:38:51.865331703 (92h9m51.736414748s) [noise=true , address=89.149.201.147:4007 , piece_id=FHFZMTRZEYLGK2QOXICD5VLT77MIFP3HL6Z7SP7URQEY5L3HIDTA, seg_pos={0 0}, upload=80368]: Long tail cancellation due to optimal threshold Dec 9 11:38:51.865403281 (92h9m51.736486326s) [seg_pos={0 0}, upload=80368]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 9 11:38:51.865634440 (92h9m51.736717485s) [upload=80368]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 11:38:51.865643307 (92h9m51.736726352s) [upload=80368]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 11:40:37.870707645 (92h11m37.741790681s) [upload=80368]: Done uploading segments... Dec 9 11:40:37.870754294 (92h11m37.741837339s) [upload=80368]: Upload finished. err: metaclient: context deadline exceeded Duration: 13.466571259s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 5.881467957s trace: https://jaeger.dev.storj.io/trace/1ad10e1389335c60 FAIL (2025-12-09 11:57:00 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.88149659s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 7.623374637s trace: https://jaeger.dev.storj.io/trace/180c8a02a447001f FAIL (2025-12-09 11:56:54 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 1m24.245559s) [upload=79956]: Starting upload Dec 9 11:10:24.374550547 (91h41m24.245633592s) [upload=79956]: Done starting upload Dec 9 11:10:24.374695774 (91h41m24.245778819s) [upload=79956]: Uploading segments... Dec 9 11:10:24.374799832 (91h41m24.245882877s) [upload=79956]: Got next segment. Inline: false Dec 9 11:10:24.374845269 (91h41m24.245928314s) [upload=79956]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 11:10:24.374891817 (91h41m24.245974862s) [upload=79956]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 11:10:24.692953929 (91h41m24.564036964s) [seg_pos={0 0}, upload=79956]: Begin upload segment... Dec 9 11:10:24.692996710 (91h41m24.564079745s) [seg_pos={0 0}, upload=79956]: Done begin upload segment. Dec 9 11:10:24.693021047 (91h41m24.564104092s) [noise=true , address=145.79.227.42:4121 , piece_id=THP22HNINRQMLRTQJKBHRCSBE5O4JMZWVNFRTJDVFCP7CKP57V7Q, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:24.693030324 (91h41m24.56411336s) [noise=true , address=145.79.227.28:4727 , piece_id=2VL43MLR2PE4BPHW4IWKNWAW3PSOAX2EV32XWLQYMYBM4FZ7RT6Q, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:24.693620949 (91h41m24.564703994s) [noise=true , address=145.79.227.23:4545 , piece_id=XVIQ424I6HQC4IXH2XAYRG7DFXJ7LJ44VT7ZCWVHFFUTXMTBZYXQ, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:24.694151741 (91h41m24.565234786s) [noise=true , address=145.79.227.41:4087 , piece_id=CC643HDKTLAEAAOHWOSXJL75BJH62EL7QISOZVG2VYPH4S3LDKMQ, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:24.694508199 (91h41m24.565591244s) [noise=true , address=145.79.227.39:4014 , piece_id=4F3Y2MMBAZVYJW6T5ZRNLQ4ZU56GF4EQMY6BPKZTCJ7KR7EY6UXQ, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:24.694727136 (91h41m24.565810181s) [noise=true , address=45.91.116.4:4939 , piece_id=QK4MQXBPU7SZ7SUA7SMDF5IJ2UUA6Q6MA2MA4UR3DSMU6QN7OHSQ, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:24.695257175 (91h41m24.56634022s) [noise=true , address=145.79.227.26:4655 , piece_id=WSNSSYE2G6LWCKNBMS63KSEPZBQSENSZ536OHOEAHKMLAIVQA4ZA, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:24.695748691 (91h41m24.566831736s) [noise=true , address=145.79.227.27:4689 , piece_id=4PFOTYEIDSKPMQF4LFVZZB5MK4O6JXFTU2QCLC5GUZTAUFPOE45Q, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:24.696362650 (91h41m24.567445695s) [noise=true , address=145.79.227.31:4799 , piece_id=4GDZXDMIOOWAIEKGEVXHGKF2LQ4YLSCCPA4S6UPCCAVHGRUT4DCA, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:24.698217953 (91h41m24.569300988s) [noise=true , address=77.90.31.14:4661 , piece_id=LJY3BTAERCAOCCIKW2THHEBAYKOEXDYQVGJTMWHYBACZJ4XJTBBA, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:24.701427525 (91h41m24.57251057s) [noise=true , address=77.90.31.12:4592 , piece_id=THMPWJ2JV2RVPMFHR4UPYJEC2FT7AS7QYVZQI427KGYKTYGI3ZMQ, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:24.705981576 (91h41m24.577064621s) [noise=true , address=145.79.227.33:4869 , piece_id=R7N7EKAQ6PUL7IBCY7SH4GTGQ6XYU3ZB7UJUVKFX5S7NTFRSCFNA, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:24.708340077 (91h41m24.579423112s) [noise=true , address=145.79.227.27:4690 , piece_id=OBIAVWCVVSLUDUW7G7ZGIQRIVOHO5R4KR25AVMHNA4RSTT6KB23Q, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:24.710358000 (91h41m24.581441035s) [noise=true , address=145.79.227.43:4157 , piece_id=7WZ2NUCKMURKNDQCZ6FHUC2I4XFXK4W3B3DR26DK6F2G62R7YC7Q, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:24.711000353 (91h41m24.582083388s) [noise=true , address=145.79.227.29:4943 , piece_id=PVDPIUVHEL3DLK45G2NTR3INZJ2CEPKLF6KX6CZMOMKXEOFRTTBA, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:24.712392574 (91h41m24.583475609s) [noise=true , address=77.90.31.16:4734 , piece_id=BQYJNHPRU7WKEHXUXJCUX6ZDU2RSVDPAAJTOD3VEQCEGE7TVR6CQ, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:26.323507311 (91h41m26.194590346s) [upload=79956]: Next returned nil segment Dec 9 11:10:26.323527019 (91h41m26.194610064s) [upload=79956]: Waiting for error group managing segments... Dec 9 11:10:26.705924029 (91h41m26.577007074s) [noise=true , address=145.79.227.28:4727 , piece_id=2VL43MLR2PE4BPHW4IWKNWAW3PSOAX2EV32XWLQYMYBM4FZ7RT6Q, seg_pos={0 0}, upload=79956]: Done uploading piece. err: <nil> Dec 9 11:10:26.713527548 (91h41m26.584610593s) [noise=true , address=145.79.227.42:4121 , piece_id=THP22HNINRQMLRTQJKBHRCSBE5O4JMZWVNFRTJDVFCP7CKP57V7Q, seg_pos={0 0}, upload=79956]: Done uploading piece. err: <nil> Dec 9 11:10:26.715957385 (91h41m26.58704043s) [noise=true , address=145.79.227.23:4545 , piece_id=XVIQ424I6HQC4IXH2XAYRG7DFXJ7LJ44VT7ZCWVHFFUTXMTBZYXQ, seg_pos={0 0}, upload=79956]: Done uploading piece. err: <nil> Dec 9 11:10:26.721154390 (91h41m26.592237435s) [noise=true , address=145.79.227.26:4655 , piece_id=WSNSSYE2G6LWCKNBMS63KSEPZBQSENSZ536OHOEAHKMLAIVQA4ZA, seg_pos={0 0}, upload=79956]: Done uploading piece. err: <nil> Dec 9 11:10:26.726209116 (91h41m26.597292161s) [noise=true , address=145.79.227.43:4157 , piece_id=7WZ2NUCKMURKNDQCZ6FHUC2I4XFXK4W3B3DR26DK6F2G62R7YC7Q, seg_pos={0 0}, upload=79956]: Done uploading piece. err: <nil> Dec 9 11:10:26.744717479 (91h41m26.615800524s) [noise=true , address=145.79.227.31:4799 , piece_id=4GDZXDMIOOWAIEKGEVXHGKF2LQ4YLSCCPA4S6UPCCAVHGRUT4DCA, seg_pos={0 0}, upload=79956]: Done uploading piece. err: <nil> Dec 9 11:10:26.865540120 (91h41m26.736623165s) [noise=true , address=77.90.31.14:4661 , piece_id=LJY3BTAERCAOCCIKW2THHEBAYKOEXDYQVGJTMWHYBACZJ4XJTBBA, seg_pos={0 0}, upload=79956]: Done uploading piece. err: <nil> Dec 9 11:10:26.874733556 (91h41m26.745816591s) [noise=true , address=77.90.31.12:4592 , piece_id=THMPWJ2JV2RVPMFHR4UPYJEC2FT7AS7QYVZQI427KGYKTYGI3ZMQ, seg_pos={0 0}, upload=79956]: Done uploading piece. err: <nil> Dec 9 11:10:27.552332367 (91h41m27.423415412s) [noise=true , address=145.79.227.39:4014 , piece_id=4F3Y2MMBAZVYJW6T5ZRNLQ4ZU56GF4EQMY6BPKZTCJ7KR7EY6UXQ, seg_pos={0 0}, upload=79956]: Done uploading piece. err: <nil> Dec 9 11:10:27.555135495 (91h41m27.42621853s) [noise=true , address=145.79.227.27:4689 , piece_id=4PFOTYEIDSKPMQF4LFVZZB5MK4O6JXFTU2QCLC5GUZTAUFPOE45Q, seg_pos={0 0}, upload=79956]: Done uploading piece. err: <nil> Dec 9 11:10:27.837239356 (91h41m27.708322401s) [noise=true , address=145.79.227.41:4087 , piece_id=CC643HDKTLAEAAOHWOSXJL75BJH62EL7QISOZVG2VYPH4S3LDKMQ, seg_pos={0 0}, upload=79956]: Done uploading piece. err: <nil> Dec 9 11:10:27.850249129 (91h41m27.721332174s) [noise=true , address=145.79.227.29:4943 , piece_id=PVDPIUVHEL3DLK45G2NTR3INZJ2CEPKLF6KX6CZMOMKXEOFRTTBA, seg_pos={0 0}, upload=79956]: Done uploading piece. err: <nil> Dec 9 11:10:27.850441505 (91h41m27.72152455s) [noise=true , address=145.79.227.33:4869 , piece_id=R7N7EKAQ6PUL7IBCY7SH4GTGQ6XYU3ZB7UJUVKFX5S7NTFRSCFNA, seg_pos={0 0}, upload=79956]: Done uploading piece. err: <nil> Dec 9 11:10:27.850450944 (91h41m27.721533989s) [seg_pos={0 0}, upload=79956]: Segment reached optimal threshold of 13 pieces. Dec 9 11:10:27.850729945 (91h41m27.72181298s) [noise=true , address=77.90.31.16:4734 , piece_id=BQYJNHPRU7WKEHXUXJCUX6ZDU2RSVDPAAJTOD3VEQCEGE7TVR6CQ, seg_pos={0 0}, upload=79956]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: write tcp 188.166.225.170:57470->77.90.31.16:4734: use of closed network connection; write tcp 188.166.225.170:57470->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57470->77.90.31.16:4734: use of closed network connection Dec 9 11:10:27.850737028 (91h41m27.721820063s) [noise=true , address=77.90.31.16:4734 , piece_id=BQYJNHPRU7WKEHXUXJCUX6ZDU2RSVDPAAJTOD3VEQCEGE7TVR6CQ, seg_pos={0 0}, upload=79956]: Long tail cancellation due to optimal threshold Dec 9 11:10:27.850945425 (91h41m27.72202848s) [noise=true , address=45.91.116.4:4939 , piece_id=QK4MQXBPU7SZ7SUA7SMDF5IJ2UUA6Q6MA2MA4UR3DSMU6QN7OHSQ, seg_pos={0 0}, upload=79956]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: read tcp 188.166.225.170:48538->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 9 11:10:27.850974751 (91h41m27.722057816s) [noise=true , address=45.91.116.4:4939 , piece_id=QK4MQXBPU7SZ7SUA7SMDF5IJ2UUA6Q6MA2MA4UR3DSMU6QN7OHSQ, seg_pos={0 0}, upload=79956]: Long tail cancellation due to optimal threshold Dec 9 11:10:27.851168249 (91h41m27.722251294s) [noise=true , address=145.79.227.27:4690 , piece_id=OBIAVWCVVSLUDUW7G7ZGIQRIVOHO5R4KR25AVMHNA4RSTT6KB23Q, seg_pos={0 0}, upload=79956]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H): protocol: expected piece hash; context canceled; EOF Dec 9 11:10:27.851178119 (91h41m27.722261194s) [noise=true , address=145.79.227.27:4690 , piece_id=OBIAVWCVVSLUDUW7G7ZGIQRIVOHO5R4KR25AVMHNA4RSTT6KB23Q, seg_pos={0 0}, upload=79956]: Long tail cancellation due to optimal threshold Dec 9 11:10:27.851202003 (91h41m27.722285048s) [seg_pos={0 0}, upload=79956]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 9 11:10:27.851316833 (91h41m27.722399908s) [upload=79956]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 11:10:27.851325489 (91h41m27.722408534s) [upload=79956]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 11:25:29.822344337 (91h56m29.693427372s) [upload=79956]: Done uploading segments... Dec 9 11:25:29.822415463 (91h56m29.693498498s) [upload=79956]: Upload finished. err: metaclient: context deadline exceeded Duration: 7.623405706s core uplink us-select-1
checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 474.165706ms remote: "tcp:35.207.121.91:443" target: https://eu1.storj.io/login trace: https://jaeger.dev.storj.io/trace/215c1ddb7fa82bd3 FAIL (2025-12-09 11:56:53 UTC) orbiter warning: Get "https://eu1.storj.io/login": EOF Duration: 474.195333ms core satellite eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 7 out of 60 is failed (threshold=6) [X...X..X............X....................XX................X] duration: 5.001633897s trace: https://jaeger.dev.storj.io/trace/36e1a86555e8f5a8 FAIL (2025-12-09 11:56:42 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001660899s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 13.05287211s trace: https://jaeger.dev.storj.io/trace/7098c007ca020436 FAIL (2025-12-09 11:56:19 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , address=185.167.204.205:4029 , piece_id=WMBEVUF62ZU76TQDICPDPMG4JA7374YQFXTI5XJFSN6MJZBATS7Q, seg_pos={0 0}, upload=80368]: Uploading piece... Dec 9 11:38:49.362641042 (92h9m49.233724077s) [noise=true , address=185.167.204.206:4037 , piece_id=LO332KWEY4NLBKVGEVME2SF7UIGTEVCKLOHRCC7FLIJEPP5DTKTA, seg_pos={0 0}, upload=80368]: Uploading piece... Dec 9 11:38:49.362747554 (92h9m49.233830599s) [noise=true , address=185.167.204.206:4045 , piece_id=7RI3JKTQDSVERWTVQ4NIGZMBNAC6OPYUCTPU77HX3V74NBG7XALQ, seg_pos={0 0}, upload=80368]: Uploading piece... Dec 9 11:38:49.363050582 (92h9m49.234133627s) [noise=true , address=185.167.204.206:4059 , piece_id=HLZEIABVICZTZCKXHBKIFQJ3UVRGVV5EWHWFFL5G3E7CYGJN4GHA, seg_pos={0 0}, upload=80368]: Uploading piece... Dec 9 11:38:49.363113251 (92h9m49.234196286s) [noise=true , address=185.167.204.205:4016 , piece_id=55J2Z5J7LPDPCBKXAQ7V4BQW4U7W6UAQVF63LYH3BPFHSEQE6CEA, seg_pos={0 0}, upload=80368]: Uploading piece... Dec 9 11:38:49.363383676 (92h9m49.234466711s) [noise=true , address=185.167.204.206:4048 , piece_id=XMV7K5EEISKMT5JQFUD6EXZ3EYELHBWF6Y3J276VUUP2YGXJKF3A, seg_pos={0 0}, upload=80368]: Uploading piece... Dec 9 11:38:49.363681573 (92h9m49.234764608s) [noise=true , address=185.167.204.205:4001 , piece_id=KIUIPM26OHXISSEHSZPLFHUXF4NVOHSY6TILREFVGDRDBB5IOCQQ, seg_pos={0 0}, upload=80368]: Uploading piece... Dec 9 11:38:49.364037821 (92h9m49.235120866s) [noise=true , address=81.171.14.70:4042 , piece_id=WWJDEHBRMRVXANEWVG6B6QLD2SR7M6FXH4642UZ3AFQSDGCO2HYA, seg_pos={0 0}, upload=80368]: Uploading piece... Dec 9 11:38:49.364321191 (92h9m49.235404226s) [noise=true , address=81.171.14.70:4032 , piece_id=GOAKND7Z3IDBUYT3ILUUBU7YC3UMA6PPDJLLMFTH2OKHF76GCHWA, seg_pos={0 0}, upload=80368]: Uploading piece... Dec 9 11:38:49.364596545 (92h9m49.23567959s) [noise=true , address=81.171.14.70:4039 , piece_id=EJYQEP7LA6QJQICJQQVTNIXDN6ID5PAPL54GADSICUBJ3YQQ5E7Q, seg_pos={0 0}, upload=80368]: Uploading piece... Dec 9 11:38:49.365194575 (92h9m49.23627762s) [noise=true , address=89.149.201.147:4011 , piece_id=IJZKML6ZS5AF7KRYSHG4ZDR7AZSJGUWRSPTLSYTWR6WKYXWORAYA, seg_pos={0 0}, upload=80368]: Uploading piece... Dec 9 11:38:50.188365108 (92h9m50.059448153s) [upload=80368]: Next returned nil segment Dec 9 11:38:50.188379346 (92h9m50.059462391s) [upload=80368]: Waiting for error group managing segments... Dec 9 11:38:50.573198910 (92h9m50.444281955s) [noise=true , address=185.167.204.206:4062 , piece_id=UZUYTLORRUJOMEX62Z7ABCOXWZHBZ7PV72MA4TZIWDDWZKBR22OA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:50.577545046 (92h9m50.448628081s) [noise=true , address=185.167.204.206:4059 , piece_id=HLZEIABVICZTZCKXHBKIFQJ3UVRGVV5EWHWFFL5G3E7CYGJN4GHA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:50.577840359 (92h9m50.448923394s) [noise=true , address=185.167.204.206:4045 , piece_id=7RI3JKTQDSVERWTVQ4NIGZMBNAC6OPYUCTPU77HX3V74NBG7XALQ, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:50.578430443 (92h9m50.449513478s) [noise=true , address=185.167.204.206:4048 , piece_id=XMV7K5EEISKMT5JQFUD6EXZ3EYELHBWF6Y3J276VUUP2YGXJKF3A, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:50.579912414 (92h9m50.450995459s) [noise=true , address=185.167.204.205:4016 , piece_id=55J2Z5J7LPDPCBKXAQ7V4BQW4U7W6UAQVF63LYH3BPFHSEQE6CEA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:50.580088579 (92h9m50.451171625s) [noise=true , address=185.167.204.206:4068 , piece_id=FDF7EQNPFGNEDO6CEPYGMAAJOIT4TPHY2PVYHNXH7ILC7AY2LO3A, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:50.582475435 (92h9m50.45355848s) [noise=true , address=185.167.204.206:4037 , piece_id=LO332KWEY4NLBKVGEVME2SF7UIGTEVCKLOHRCC7FLIJEPP5DTKTA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:50.586259020 (92h9m50.457342055s) [noise=true , address=185.167.204.205:4030 , piece_id=VHBCZ6QQRBZ4THX6XCFIZ6JSFMIONQFSEZ3ERGOZB6L7L5USAKMA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:50.586458941 (92h9m50.457541986s) [noise=true , address=185.167.204.206:4060 , piece_id=A3NTK6DGBBOACMUONBGII27GVS7QZWOGDEKBPZFX76RSH5HMXS5A, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:50.588436907 (92h9m50.459519942s) [noise=true , address=185.167.204.205:4009 , piece_id=5WUDBEGUENMGEMSND2VCENBNGG2BIV47ICSWN3CJYLQQ3XIVZXKQ, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:50.589577038 (92h9m50.460660083s) [noise=true , address=185.167.204.205:4029 , piece_id=WMBEVUF62ZU76TQDICPDPMG4JA7374YQFXTI5XJFSN6MJZBATS7Q, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:50.592840462 (92h9m50.463923507s) [noise=true , address=185.167.204.205:4001 , piece_id=KIUIPM26OHXISSEHSZPLFHUXF4NVOHSY6TILREFVGDRDBB5IOCQQ, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.141501915 (92h9m51.01258496s) [noise=true , address=81.171.14.70:4039 , piece_id=EJYQEP7LA6QJQICJQQVTNIXDN6ID5PAPL54GADSICUBJ3YQQ5E7Q, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.153777871 (92h9m51.024860916s) [noise=true , address=81.171.14.70:4042 , piece_id=WWJDEHBRMRVXANEWVG6B6QLD2SR7M6FXH4642UZ3AFQSDGCO2HYA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.153948696 (92h9m51.025031751s) [noise=true , address=81.171.14.70:4028 , piece_id=NSYGSKJBIHMHZZX7HK7N3UQDGKWWQARJPGLAWH2TEPGWKNYWPC4A, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.157888027 (92h9m51.028971062s) [noise=true , address=81.171.14.70:4041 , piece_id=PZNQJLGVYH3CBPX4L2URCEVLMGK4JLQAMVFX25UBONKVYSTSEEWQ, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.170799434 (92h9m51.041882489s) [noise=true , address=81.171.14.70:4045 , piece_id=TZIZNHRR66JMWMZJNUHYFPYJ2ORR3OLKZ7VOQ5MIWXQOFARHL4VA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.171007360 (92h9m51.042090405s) [noise=true , address=81.171.14.70:4035 , piece_id=5T5MX4YDMI4IUY4LDSREN2MTN3NUL4GS6KZXDMNZMKJYY3VMS47Q, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.184999573 (92h9m51.056082648s) [noise=true , address=81.171.14.70:4032 , piece_id=GOAKND7Z3IDBUYT3ILUUBU7YC3UMA6PPDJLLMFTH2OKHF76GCHWA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.196460517 (92h9m51.067543562s) [noise=true , address=81.171.14.70:4040 , piece_id=4GA5CWKM5RJBL2IMWZNRGAA2T2JTFX54AFEFI3NAV2GNTKIDUELQ, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.199311115 (92h9m51.07039416s) [noise=true , address=81.171.14.70:4037 , piece_id=G6MBAEOMB7R4MQKZDULZY4DYQ5NR2QMCKW6HWYA55VD5YMSAERPA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.237519992 (92h9m51.108603028s) [noise=true , address=81.171.14.70:4030 , piece_id=VFAGZYZMOZKEHRNPUR2D2JCRDZFUFXUOS3JTTI2UOUYTIUY74NFQ, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.733107833 (92h9m51.604190868s) [noise=true , address=89.149.201.147:4005 , piece_id=OCHDE5VQIQPUVCOFXX343C7LXUIJFD4Q62ZZPEXKRN63KARUQR5A, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.815947736 (92h9m51.687030771s) [noise=true , address=89.149.201.147:4015 , piece_id=PGXJFJBHXVMXBTIWDTAOZKAU2JKTFSGCAD5I5MWU2MD32GF3AMDQ, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.841035780 (92h9m51.712118825s) [noise=true , address=89.149.201.147:4020 , piece_id=KIYFOE4WT3CGQQ3CL3BP2A3HH2KVKKLAUUGMHYPADVKI7BVBBRTA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.864720530 (92h9m51.735803575s) [noise=true , address=89.149.201.147:4002 , piece_id=TIK5YZA7N6MYGZ6KGNXPFBEQXD3DE2UXQ5Q2JP3EZGSLJ7TAO2GA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.864741660 (92h9m51.735824705s) [seg_pos={0 0}, upload=80368]: Segment reached optimal threshold of 26 pieces. Dec 9 11:38:51.865018136 (92h9m51.736101181s) [noise=true , address=89.149.201.147:4011 , piece_id=IJZKML6ZS5AF7KRYSHG4ZDR7AZSJGUWRSPTLSYTWR6WKYXWORAYA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PcxodnQMJERGjvBgnz3iEiC9MseRhMydkCriwGpQKqi6DGFDg): protocol: expected piece hash; context canceled; EOF Dec 9 11:38:51.865029489 (92h9m51.736112534s) [noise=true , address=89.149.201.147:4011 , piece_id=IJZKML6ZS5AF7KRYSHG4ZDR7AZSJGUWRSPTLSYTWR6WKYXWORAYA, seg_pos={0 0}, upload=80368]: Long tail cancellation due to optimal threshold Dec 9 11:38:51.865308640 (92h9m51.736391685s) [noise=true , address=89.149.201.147:4007 , piece_id=FHFZMTRZEYLGK2QOXICD5VLT77MIFP3HL6Z7SP7URQEY5L3HIDTA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Q7d4vj1KoWhDr6xd9gEfFrZ1sV3Yro2AFke2f3Ke1rXsGnvew): protocol: expected piece hash; context canceled; EOF Dec 9 11:38:51.865331703 (92h9m51.736414748s) [noise=true , address=89.149.201.147:4007 , piece_id=FHFZMTRZEYLGK2QOXICD5VLT77MIFP3HL6Z7SP7URQEY5L3HIDTA, seg_pos={0 0}, upload=80368]: Long tail cancellation due to optimal threshold Dec 9 11:38:51.865403281 (92h9m51.736486326s) [seg_pos={0 0}, upload=80368]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 9 11:38:51.865634440 (92h9m51.736717485s) [upload=80368]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 11:38:51.865643307 (92h9m51.736726352s) [upload=80368]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 11:40:37.870707645 (92h11m37.741790681s) [upload=80368]: Done uploading segments... Dec 9 11:40:37.870754294 (92h11m37.741837339s) [upload=80368]: Upload finished. err: metaclient: context deadline exceeded Duration: 13.052904862s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 7.415842644s trace: https://jaeger.dev.storj.io/trace/180c8a02a447001f FAIL (2025-12-09 11:56:14 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 1m24.245559s) [upload=79956]: Starting upload Dec 9 11:10:24.374550547 (91h41m24.245633592s) [upload=79956]: Done starting upload Dec 9 11:10:24.374695774 (91h41m24.245778819s) [upload=79956]: Uploading segments... Dec 9 11:10:24.374799832 (91h41m24.245882877s) [upload=79956]: Got next segment. Inline: false Dec 9 11:10:24.374845269 (91h41m24.245928314s) [upload=79956]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 11:10:24.374891817 (91h41m24.245974862s) [upload=79956]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 11:10:24.692953929 (91h41m24.564036964s) [seg_pos={0 0}, upload=79956]: Begin upload segment... Dec 9 11:10:24.692996710 (91h41m24.564079745s) [seg_pos={0 0}, upload=79956]: Done begin upload segment. Dec 9 11:10:24.693021047 (91h41m24.564104092s) [noise=true , address=145.79.227.42:4121 , piece_id=THP22HNINRQMLRTQJKBHRCSBE5O4JMZWVNFRTJDVFCP7CKP57V7Q, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:24.693030324 (91h41m24.56411336s) [noise=true , address=145.79.227.28:4727 , piece_id=2VL43MLR2PE4BPHW4IWKNWAW3PSOAX2EV32XWLQYMYBM4FZ7RT6Q, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:24.693620949 (91h41m24.564703994s) [noise=true , address=145.79.227.23:4545 , piece_id=XVIQ424I6HQC4IXH2XAYRG7DFXJ7LJ44VT7ZCWVHFFUTXMTBZYXQ, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:24.694151741 (91h41m24.565234786s) [noise=true , address=145.79.227.41:4087 , piece_id=CC643HDKTLAEAAOHWOSXJL75BJH62EL7QISOZVG2VYPH4S3LDKMQ, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:24.694508199 (91h41m24.565591244s) [noise=true , address=145.79.227.39:4014 , piece_id=4F3Y2MMBAZVYJW6T5ZRNLQ4ZU56GF4EQMY6BPKZTCJ7KR7EY6UXQ, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:24.694727136 (91h41m24.565810181s) [noise=true , address=45.91.116.4:4939 , piece_id=QK4MQXBPU7SZ7SUA7SMDF5IJ2UUA6Q6MA2MA4UR3DSMU6QN7OHSQ, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:24.695257175 (91h41m24.56634022s) [noise=true , address=145.79.227.26:4655 , piece_id=WSNSSYE2G6LWCKNBMS63KSEPZBQSENSZ536OHOEAHKMLAIVQA4ZA, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:24.695748691 (91h41m24.566831736s) [noise=true , address=145.79.227.27:4689 , piece_id=4PFOTYEIDSKPMQF4LFVZZB5MK4O6JXFTU2QCLC5GUZTAUFPOE45Q, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:24.696362650 (91h41m24.567445695s) [noise=true , address=145.79.227.31:4799 , piece_id=4GDZXDMIOOWAIEKGEVXHGKF2LQ4YLSCCPA4S6UPCCAVHGRUT4DCA, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:24.698217953 (91h41m24.569300988s) [noise=true , address=77.90.31.14:4661 , piece_id=LJY3BTAERCAOCCIKW2THHEBAYKOEXDYQVGJTMWHYBACZJ4XJTBBA, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:24.701427525 (91h41m24.57251057s) [noise=true , address=77.90.31.12:4592 , piece_id=THMPWJ2JV2RVPMFHR4UPYJEC2FT7AS7QYVZQI427KGYKTYGI3ZMQ, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:24.705981576 (91h41m24.577064621s) [noise=true , address=145.79.227.33:4869 , piece_id=R7N7EKAQ6PUL7IBCY7SH4GTGQ6XYU3ZB7UJUVKFX5S7NTFRSCFNA, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:24.708340077 (91h41m24.579423112s) [noise=true , address=145.79.227.27:4690 , piece_id=OBIAVWCVVSLUDUW7G7ZGIQRIVOHO5R4KR25AVMHNA4RSTT6KB23Q, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:24.710358000 (91h41m24.581441035s) [noise=true , address=145.79.227.43:4157 , piece_id=7WZ2NUCKMURKNDQCZ6FHUC2I4XFXK4W3B3DR26DK6F2G62R7YC7Q, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:24.711000353 (91h41m24.582083388s) [noise=true , address=145.79.227.29:4943 , piece_id=PVDPIUVHEL3DLK45G2NTR3INZJ2CEPKLF6KX6CZMOMKXEOFRTTBA, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:24.712392574 (91h41m24.583475609s) [noise=true , address=77.90.31.16:4734 , piece_id=BQYJNHPRU7WKEHXUXJCUX6ZDU2RSVDPAAJTOD3VEQCEGE7TVR6CQ, seg_pos={0 0}, upload=79956]: Uploading piece... Dec 9 11:10:26.323507311 (91h41m26.194590346s) [upload=79956]: Next returned nil segment Dec 9 11:10:26.323527019 (91h41m26.194610064s) [upload=79956]: Waiting for error group managing segments... Dec 9 11:10:26.705924029 (91h41m26.577007074s) [noise=true , address=145.79.227.28:4727 , piece_id=2VL43MLR2PE4BPHW4IWKNWAW3PSOAX2EV32XWLQYMYBM4FZ7RT6Q, seg_pos={0 0}, upload=79956]: Done uploading piece. err: <nil> Dec 9 11:10:26.713527548 (91h41m26.584610593s) [noise=true , address=145.79.227.42:4121 , piece_id=THP22HNINRQMLRTQJKBHRCSBE5O4JMZWVNFRTJDVFCP7CKP57V7Q, seg_pos={0 0}, upload=79956]: Done uploading piece. err: <nil> Dec 9 11:10:26.715957385 (91h41m26.58704043s) [noise=true , address=145.79.227.23:4545 , piece_id=XVIQ424I6HQC4IXH2XAYRG7DFXJ7LJ44VT7ZCWVHFFUTXMTBZYXQ, seg_pos={0 0}, upload=79956]: Done uploading piece. err: <nil> Dec 9 11:10:26.721154390 (91h41m26.592237435s) [noise=true , address=145.79.227.26:4655 , piece_id=WSNSSYE2G6LWCKNBMS63KSEPZBQSENSZ536OHOEAHKMLAIVQA4ZA, seg_pos={0 0}, upload=79956]: Done uploading piece. err: <nil> Dec 9 11:10:26.726209116 (91h41m26.597292161s) [noise=true , address=145.79.227.43:4157 , piece_id=7WZ2NUCKMURKNDQCZ6FHUC2I4XFXK4W3B3DR26DK6F2G62R7YC7Q, seg_pos={0 0}, upload=79956]: Done uploading piece. err: <nil> Dec 9 11:10:26.744717479 (91h41m26.615800524s) [noise=true , address=145.79.227.31:4799 , piece_id=4GDZXDMIOOWAIEKGEVXHGKF2LQ4YLSCCPA4S6UPCCAVHGRUT4DCA, seg_pos={0 0}, upload=79956]: Done uploading piece. err: <nil> Dec 9 11:10:26.865540120 (91h41m26.736623165s) [noise=true , address=77.90.31.14:4661 , piece_id=LJY3BTAERCAOCCIKW2THHEBAYKOEXDYQVGJTMWHYBACZJ4XJTBBA, seg_pos={0 0}, upload=79956]: Done uploading piece. err: <nil> Dec 9 11:10:26.874733556 (91h41m26.745816591s) [noise=true , address=77.90.31.12:4592 , piece_id=THMPWJ2JV2RVPMFHR4UPYJEC2FT7AS7QYVZQI427KGYKTYGI3ZMQ, seg_pos={0 0}, upload=79956]: Done uploading piece. err: <nil> Dec 9 11:10:27.552332367 (91h41m27.423415412s) [noise=true , address=145.79.227.39:4014 , piece_id=4F3Y2MMBAZVYJW6T5ZRNLQ4ZU56GF4EQMY6BPKZTCJ7KR7EY6UXQ, seg_pos={0 0}, upload=79956]: Done uploading piece. err: <nil> Dec 9 11:10:27.555135495 (91h41m27.42621853s) [noise=true , address=145.79.227.27:4689 , piece_id=4PFOTYEIDSKPMQF4LFVZZB5MK4O6JXFTU2QCLC5GUZTAUFPOE45Q, seg_pos={0 0}, upload=79956]: Done uploading piece. err: <nil> Dec 9 11:10:27.837239356 (91h41m27.708322401s) [noise=true , address=145.79.227.41:4087 , piece_id=CC643HDKTLAEAAOHWOSXJL75BJH62EL7QISOZVG2VYPH4S3LDKMQ, seg_pos={0 0}, upload=79956]: Done uploading piece. err: <nil> Dec 9 11:10:27.850249129 (91h41m27.721332174s) [noise=true , address=145.79.227.29:4943 , piece_id=PVDPIUVHEL3DLK45G2NTR3INZJ2CEPKLF6KX6CZMOMKXEOFRTTBA, seg_pos={0 0}, upload=79956]: Done uploading piece. err: <nil> Dec 9 11:10:27.850441505 (91h41m27.72152455s) [noise=true , address=145.79.227.33:4869 , piece_id=R7N7EKAQ6PUL7IBCY7SH4GTGQ6XYU3ZB7UJUVKFX5S7NTFRSCFNA, seg_pos={0 0}, upload=79956]: Done uploading piece. err: <nil> Dec 9 11:10:27.850450944 (91h41m27.721533989s) [seg_pos={0 0}, upload=79956]: Segment reached optimal threshold of 13 pieces. Dec 9 11:10:27.850729945 (91h41m27.72181298s) [noise=true , address=77.90.31.16:4734 , piece_id=BQYJNHPRU7WKEHXUXJCUX6ZDU2RSVDPAAJTOD3VEQCEGE7TVR6CQ, seg_pos={0 0}, upload=79956]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: write tcp 188.166.225.170:57470->77.90.31.16:4734: use of closed network connection; write tcp 188.166.225.170:57470->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57470->77.90.31.16:4734: use of closed network connection Dec 9 11:10:27.850737028 (91h41m27.721820063s) [noise=true , address=77.90.31.16:4734 , piece_id=BQYJNHPRU7WKEHXUXJCUX6ZDU2RSVDPAAJTOD3VEQCEGE7TVR6CQ, seg_pos={0 0}, upload=79956]: Long tail cancellation due to optimal threshold Dec 9 11:10:27.850945425 (91h41m27.72202848s) [noise=true , address=45.91.116.4:4939 , piece_id=QK4MQXBPU7SZ7SUA7SMDF5IJ2UUA6Q6MA2MA4UR3DSMU6QN7OHSQ, seg_pos={0 0}, upload=79956]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: read tcp 188.166.225.170:48538->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 9 11:10:27.850974751 (91h41m27.722057816s) [noise=true , address=45.91.116.4:4939 , piece_id=QK4MQXBPU7SZ7SUA7SMDF5IJ2UUA6Q6MA2MA4UR3DSMU6QN7OHSQ, seg_pos={0 0}, upload=79956]: Long tail cancellation due to optimal threshold Dec 9 11:10:27.851168249 (91h41m27.722251294s) [noise=true , address=145.79.227.27:4690 , piece_id=OBIAVWCVVSLUDUW7G7ZGIQRIVOHO5R4KR25AVMHNA4RSTT6KB23Q, seg_pos={0 0}, upload=79956]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H): protocol: expected piece hash; context canceled; EOF Dec 9 11:10:27.851178119 (91h41m27.722261194s) [noise=true , address=145.79.227.27:4690 , piece_id=OBIAVWCVVSLUDUW7G7ZGIQRIVOHO5R4KR25AVMHNA4RSTT6KB23Q, seg_pos={0 0}, upload=79956]: Long tail cancellation due to optimal threshold Dec 9 11:10:27.851202003 (91h41m27.722285048s) [seg_pos={0 0}, upload=79956]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 9 11:10:27.851316833 (91h41m27.722399908s) [upload=79956]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 11:10:27.851325489 (91h41m27.722408534s) [upload=79956]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 11:25:29.822344337 (91h56m29.693427372s) [upload=79956]: Done uploading segments... Dec 9 11:25:29.822415463 (91h56m29.693498498s) [upload=79956]: Upload finished. err: metaclient: context deadline exceeded Duration: 7.415874565s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 6.036965741s trace: https://jaeger.dev.storj.io/trace/1ad10e1389335c60 FAIL (2025-12-09 11:56:05 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.036991891s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 13.188378466s trace: https://jaeger.dev.storj.io/trace/7098c007ca020436 FAIL (2025-12-09 11:56:00 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , address=185.167.204.205:4029 , piece_id=WMBEVUF62ZU76TQDICPDPMG4JA7374YQFXTI5XJFSN6MJZBATS7Q, seg_pos={0 0}, upload=80368]: Uploading piece... Dec 9 11:38:49.362641042 (92h9m49.233724077s) [noise=true , address=185.167.204.206:4037 , piece_id=LO332KWEY4NLBKVGEVME2SF7UIGTEVCKLOHRCC7FLIJEPP5DTKTA, seg_pos={0 0}, upload=80368]: Uploading piece... Dec 9 11:38:49.362747554 (92h9m49.233830599s) [noise=true , address=185.167.204.206:4045 , piece_id=7RI3JKTQDSVERWTVQ4NIGZMBNAC6OPYUCTPU77HX3V74NBG7XALQ, seg_pos={0 0}, upload=80368]: Uploading piece... Dec 9 11:38:49.363050582 (92h9m49.234133627s) [noise=true , address=185.167.204.206:4059 , piece_id=HLZEIABVICZTZCKXHBKIFQJ3UVRGVV5EWHWFFL5G3E7CYGJN4GHA, seg_pos={0 0}, upload=80368]: Uploading piece... Dec 9 11:38:49.363113251 (92h9m49.234196286s) [noise=true , address=185.167.204.205:4016 , piece_id=55J2Z5J7LPDPCBKXAQ7V4BQW4U7W6UAQVF63LYH3BPFHSEQE6CEA, seg_pos={0 0}, upload=80368]: Uploading piece... Dec 9 11:38:49.363383676 (92h9m49.234466711s) [noise=true , address=185.167.204.206:4048 , piece_id=XMV7K5EEISKMT5JQFUD6EXZ3EYELHBWF6Y3J276VUUP2YGXJKF3A, seg_pos={0 0}, upload=80368]: Uploading piece... Dec 9 11:38:49.363681573 (92h9m49.234764608s) [noise=true , address=185.167.204.205:4001 , piece_id=KIUIPM26OHXISSEHSZPLFHUXF4NVOHSY6TILREFVGDRDBB5IOCQQ, seg_pos={0 0}, upload=80368]: Uploading piece... Dec 9 11:38:49.364037821 (92h9m49.235120866s) [noise=true , address=81.171.14.70:4042 , piece_id=WWJDEHBRMRVXANEWVG6B6QLD2SR7M6FXH4642UZ3AFQSDGCO2HYA, seg_pos={0 0}, upload=80368]: Uploading piece... Dec 9 11:38:49.364321191 (92h9m49.235404226s) [noise=true , address=81.171.14.70:4032 , piece_id=GOAKND7Z3IDBUYT3ILUUBU7YC3UMA6PPDJLLMFTH2OKHF76GCHWA, seg_pos={0 0}, upload=80368]: Uploading piece... Dec 9 11:38:49.364596545 (92h9m49.23567959s) [noise=true , address=81.171.14.70:4039 , piece_id=EJYQEP7LA6QJQICJQQVTNIXDN6ID5PAPL54GADSICUBJ3YQQ5E7Q, seg_pos={0 0}, upload=80368]: Uploading piece... Dec 9 11:38:49.365194575 (92h9m49.23627762s) [noise=true , address=89.149.201.147:4011 , piece_id=IJZKML6ZS5AF7KRYSHG4ZDR7AZSJGUWRSPTLSYTWR6WKYXWORAYA, seg_pos={0 0}, upload=80368]: Uploading piece... Dec 9 11:38:50.188365108 (92h9m50.059448153s) [upload=80368]: Next returned nil segment Dec 9 11:38:50.188379346 (92h9m50.059462391s) [upload=80368]: Waiting for error group managing segments... Dec 9 11:38:50.573198910 (92h9m50.444281955s) [noise=true , address=185.167.204.206:4062 , piece_id=UZUYTLORRUJOMEX62Z7ABCOXWZHBZ7PV72MA4TZIWDDWZKBR22OA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:50.577545046 (92h9m50.448628081s) [noise=true , address=185.167.204.206:4059 , piece_id=HLZEIABVICZTZCKXHBKIFQJ3UVRGVV5EWHWFFL5G3E7CYGJN4GHA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:50.577840359 (92h9m50.448923394s) [noise=true , address=185.167.204.206:4045 , piece_id=7RI3JKTQDSVERWTVQ4NIGZMBNAC6OPYUCTPU77HX3V74NBG7XALQ, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:50.578430443 (92h9m50.449513478s) [noise=true , address=185.167.204.206:4048 , piece_id=XMV7K5EEISKMT5JQFUD6EXZ3EYELHBWF6Y3J276VUUP2YGXJKF3A, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:50.579912414 (92h9m50.450995459s) [noise=true , address=185.167.204.205:4016 , piece_id=55J2Z5J7LPDPCBKXAQ7V4BQW4U7W6UAQVF63LYH3BPFHSEQE6CEA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:50.580088579 (92h9m50.451171625s) [noise=true , address=185.167.204.206:4068 , piece_id=FDF7EQNPFGNEDO6CEPYGMAAJOIT4TPHY2PVYHNXH7ILC7AY2LO3A, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:50.582475435 (92h9m50.45355848s) [noise=true , address=185.167.204.206:4037 , piece_id=LO332KWEY4NLBKVGEVME2SF7UIGTEVCKLOHRCC7FLIJEPP5DTKTA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:50.586259020 (92h9m50.457342055s) [noise=true , address=185.167.204.205:4030 , piece_id=VHBCZ6QQRBZ4THX6XCFIZ6JSFMIONQFSEZ3ERGOZB6L7L5USAKMA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:50.586458941 (92h9m50.457541986s) [noise=true , address=185.167.204.206:4060 , piece_id=A3NTK6DGBBOACMUONBGII27GVS7QZWOGDEKBPZFX76RSH5HMXS5A, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:50.588436907 (92h9m50.459519942s) [noise=true , address=185.167.204.205:4009 , piece_id=5WUDBEGUENMGEMSND2VCENBNGG2BIV47ICSWN3CJYLQQ3XIVZXKQ, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:50.589577038 (92h9m50.460660083s) [noise=true , address=185.167.204.205:4029 , piece_id=WMBEVUF62ZU76TQDICPDPMG4JA7374YQFXTI5XJFSN6MJZBATS7Q, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:50.592840462 (92h9m50.463923507s) [noise=true , address=185.167.204.205:4001 , piece_id=KIUIPM26OHXISSEHSZPLFHUXF4NVOHSY6TILREFVGDRDBB5IOCQQ, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.141501915 (92h9m51.01258496s) [noise=true , address=81.171.14.70:4039 , piece_id=EJYQEP7LA6QJQICJQQVTNIXDN6ID5PAPL54GADSICUBJ3YQQ5E7Q, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.153777871 (92h9m51.024860916s) [noise=true , address=81.171.14.70:4042 , piece_id=WWJDEHBRMRVXANEWVG6B6QLD2SR7M6FXH4642UZ3AFQSDGCO2HYA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.153948696 (92h9m51.025031751s) [noise=true , address=81.171.14.70:4028 , piece_id=NSYGSKJBIHMHZZX7HK7N3UQDGKWWQARJPGLAWH2TEPGWKNYWPC4A, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.157888027 (92h9m51.028971062s) [noise=true , address=81.171.14.70:4041 , piece_id=PZNQJLGVYH3CBPX4L2URCEVLMGK4JLQAMVFX25UBONKVYSTSEEWQ, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.170799434 (92h9m51.041882489s) [noise=true , address=81.171.14.70:4045 , piece_id=TZIZNHRR66JMWMZJNUHYFPYJ2ORR3OLKZ7VOQ5MIWXQOFARHL4VA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.171007360 (92h9m51.042090405s) [noise=true , address=81.171.14.70:4035 , piece_id=5T5MX4YDMI4IUY4LDSREN2MTN3NUL4GS6KZXDMNZMKJYY3VMS47Q, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.184999573 (92h9m51.056082648s) [noise=true , address=81.171.14.70:4032 , piece_id=GOAKND7Z3IDBUYT3ILUUBU7YC3UMA6PPDJLLMFTH2OKHF76GCHWA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.196460517 (92h9m51.067543562s) [noise=true , address=81.171.14.70:4040 , piece_id=4GA5CWKM5RJBL2IMWZNRGAA2T2JTFX54AFEFI3NAV2GNTKIDUELQ, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.199311115 (92h9m51.07039416s) [noise=true , address=81.171.14.70:4037 , piece_id=G6MBAEOMB7R4MQKZDULZY4DYQ5NR2QMCKW6HWYA55VD5YMSAERPA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.237519992 (92h9m51.108603028s) [noise=true , address=81.171.14.70:4030 , piece_id=VFAGZYZMOZKEHRNPUR2D2JCRDZFUFXUOS3JTTI2UOUYTIUY74NFQ, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.733107833 (92h9m51.604190868s) [noise=true , address=89.149.201.147:4005 , piece_id=OCHDE5VQIQPUVCOFXX343C7LXUIJFD4Q62ZZPEXKRN63KARUQR5A, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.815947736 (92h9m51.687030771s) [noise=true , address=89.149.201.147:4015 , piece_id=PGXJFJBHXVMXBTIWDTAOZKAU2JKTFSGCAD5I5MWU2MD32GF3AMDQ, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.841035780 (92h9m51.712118825s) [noise=true , address=89.149.201.147:4020 , piece_id=KIYFOE4WT3CGQQ3CL3BP2A3HH2KVKKLAUUGMHYPADVKI7BVBBRTA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.864720530 (92h9m51.735803575s) [noise=true , address=89.149.201.147:4002 , piece_id=TIK5YZA7N6MYGZ6KGNXPFBEQXD3DE2UXQ5Q2JP3EZGSLJ7TAO2GA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.864741660 (92h9m51.735824705s) [seg_pos={0 0}, upload=80368]: Segment reached optimal threshold of 26 pieces. Dec 9 11:38:51.865018136 (92h9m51.736101181s) [noise=true , address=89.149.201.147:4011 , piece_id=IJZKML6ZS5AF7KRYSHG4ZDR7AZSJGUWRSPTLSYTWR6WKYXWORAYA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PcxodnQMJERGjvBgnz3iEiC9MseRhMydkCriwGpQKqi6DGFDg): protocol: expected piece hash; context canceled; EOF Dec 9 11:38:51.865029489 (92h9m51.736112534s) [noise=true , address=89.149.201.147:4011 , piece_id=IJZKML6ZS5AF7KRYSHG4ZDR7AZSJGUWRSPTLSYTWR6WKYXWORAYA, seg_pos={0 0}, upload=80368]: Long tail cancellation due to optimal threshold Dec 9 11:38:51.865308640 (92h9m51.736391685s) [noise=true , address=89.149.201.147:4007 , piece_id=FHFZMTRZEYLGK2QOXICD5VLT77MIFP3HL6Z7SP7URQEY5L3HIDTA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Q7d4vj1KoWhDr6xd9gEfFrZ1sV3Yro2AFke2f3Ke1rXsGnvew): protocol: expected piece hash; context canceled; EOF Dec 9 11:38:51.865331703 (92h9m51.736414748s) [noise=true , address=89.149.201.147:4007 , piece_id=FHFZMTRZEYLGK2QOXICD5VLT77MIFP3HL6Z7SP7URQEY5L3HIDTA, seg_pos={0 0}, upload=80368]: Long tail cancellation due to optimal threshold Dec 9 11:38:51.865403281 (92h9m51.736486326s) [seg_pos={0 0}, upload=80368]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 9 11:38:51.865634440 (92h9m51.736717485s) [upload=80368]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 11:38:51.865643307 (92h9m51.736726352s) [upload=80368]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 11:40:37.870707645 (92h11m37.741790681s) [upload=80368]: Done uploading segments... Dec 9 11:40:37.870754294 (92h11m37.741837339s) [upload=80368]: Upload finished. err: metaclient: context deadline exceeded Duration: 13.188495087s core uplink eu-select-1 eu1-placement-15

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login)

checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login)

checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login)

checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)