Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [.........................X...................X.......X.X....] duration: 53.280039417s trace: https://jaeger.dev.storj.io/trace/262a3b4a3a5f2b2e FAIL (2025-12-29 03:50:36 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== se=true , address=64.49.13.27:4958 , piece_id=RNO7Q6EBDZOFRIDD6M4BU53OXVB52P3ZQZKC7ZQ6VEOD2AIO6MEQ, seg_pos={0 0}, upload=494748]: Uploading piece... Dec 29 03:46:39.389590023 (564h17m39.260673058s) [noise=true , address=77.90.31.11:4581 , piece_id=STAAU2BS7L5JZ2E6HW65EKZ2LTJTHB2BQT3E5IQPZ6PMFTB5R2BA, seg_pos={0 0}, upload=494748]: Uploading piece... Dec 29 03:46:39.389892929 (564h17m39.260975964s) [noise=true , address=77.90.31.16:4756 , piece_id=VH4O5TLJB2E2CI7RVAIB77AUU7T4Z7PQCFNHAHRHMWRK5U3SAU7A, seg_pos={0 0}, upload=494748]: Uploading piece... Dec 29 03:46:39.390155760 (564h17m39.261238805s) [noise=true , address=5.231.73.3:4023 , piece_id=OUGY6RHELOVD42KM3FMGBRXKRKGOUOHPY3DKSTJWGTUJ3Q5YROLQ, seg_pos={0 0}, upload=494748]: Uploading piece... Dec 29 03:46:39.390429110 (564h17m39.261512145s) [noise=true , address=145.79.227.44:4208 , piece_id=ZMYK4NBXCN3LAVJHAEUFIGIPQRPQ4IFLCPKN52XDNPQ3CKGE77NQ, seg_pos={0 0}, upload=494748]: Uploading piece... Dec 29 03:46:39.390774769 (564h17m39.261857804s) [noise=true , address=64.49.13.169:4710 , piece_id=Q27PBKCIHI36B5HIDKD6PKYVWTSI7QNV3CNEKVSSBKN3FVTPFDQA, seg_pos={0 0}, upload=494748]: Uploading piece... Dec 29 03:46:39.391067896 (564h17m39.262150931s) [noise=true , address=145.79.227.40:4068 , piece_id=6RC5TFIDPFVPJXY4QNUX4GVDCR5562GP6J5IMVMWTQKUVUVC4XAQ, seg_pos={0 0}, upload=494748]: Uploading piece... Dec 29 03:46:39.391349352 (564h17m39.262432397s) [noise=true , address=77.90.31.14:4673 , piece_id=JQMMLQPK3NY4XLMX3G4DZUV7WEFNXDWGGJ4FH3BFC5YPHYWK5L4A, seg_pos={0 0}, upload=494748]: Uploading piece... Dec 29 03:46:39.391620058 (564h17m39.262703113s) [noise=true , address=145.79.227.30:4779 , piece_id=FTSLUGXEJXOUE7MEFZWD6KJFAV6RGPWS7N7OJFSHADZZ7IIX55GQ, seg_pos={0 0}, upload=494748]: Uploading piece... Dec 29 03:46:39.391987468 (564h17m39.263070503s) [noise=true , address=77.90.31.13:4639 , piece_id=HMW2XHEVCPMULPKO7OUJYOU5BETJIXQVEXZSTY2KARXOD7HXT26Q, seg_pos={0 0}, upload=494748]: Uploading piece... Dec 29 03:46:39.392352663 (564h17m39.263435698s) [noise=true , address=77.90.31.18:4814 , piece_id=VUWCBNWEBOQP22BQVQ6M6KAHKOYJZNUIOB4ZYUZQDI7L4672NW7Q, seg_pos={0 0}, upload=494748]: Uploading piece... Dec 29 03:46:39.393422620 (564h17m39.264505656s) [noise=true , address=77.90.31.12:4604 , piece_id=FQDKYUMG2XHD4FQXGXZDTXG6MMTLNZYWDC47JJGUIOEXYTEYBMGA, seg_pos={0 0}, upload=494748]: Uploading piece... Dec 29 03:46:39.394657942 (564h17m39.265740977s) [noise=true , address=145.79.227.25:4652 , piece_id=S2HK4O5A2SDILSDE2YW6HSHC5JGD4FHMFR6GRWP6WEZJSXX6JNQA, seg_pos={0 0}, upload=494748]: Uploading piece... Dec 29 03:46:39.395699916 (564h17m39.266782951s) [noise=true , address=145.79.227.34:4925 , piece_id=XKSPDMDB2OUTOJVDJWJPCBBRLOHCL2NCKUBWK4IC43CWFC5LIJFA, seg_pos={0 0}, upload=494748]: Uploading piece... Dec 29 03:46:39.396753092 (564h17m39.267836127s) [noise=true , address=193.149.164.143:4699 , piece_id=BPV626GYTLB5KDERSZ5YMLKZ6MDJWOPK5MTXNJ4J4FU7PNHOCKIQ, seg_pos={0 0}, upload=494748]: Uploading piece... Dec 29 03:46:39.398134963 (564h17m39.269217998s) [noise=true , address=145.79.227.12:4773 , piece_id=L2O453KQ6JWGDEJT2K4VA4M2MVBSHKDJKNE7UASDJ5KPZNH4L2HA, seg_pos={0 0}, upload=494748]: Uploading piece... Dec 29 03:46:40.716827029 (564h17m40.587910064s) [upload=494748]: Next returned nil segment Dec 29 03:46:40.716839693 (564h17m40.587922728s) [upload=494748]: Waiting for error group managing segments... Dec 29 03:46:41.051033808 (564h17m40.922116843s) [noise=true , address=64.49.13.27:4958 , piece_id=RNO7Q6EBDZOFRIDD6M4BU53OXVB52P3ZQZKC7ZQ6VEOD2AIO6MEQ, seg_pos={0 0}, upload=494748]: Done uploading piece. err: <nil> Dec 29 03:46:41.262360402 (564h17m41.133443447s) [noise=true , address=145.79.227.12:4773 , piece_id=L2O453KQ6JWGDEJT2K4VA4M2MVBSHKDJKNE7UASDJ5KPZNH4L2HA, seg_pos={0 0}, upload=494748]: Done uploading piece. err: <nil> Dec 29 03:46:41.265241008 (564h17m41.136324063s) [noise=true , address=145.79.227.25:4652 , piece_id=S2HK4O5A2SDILSDE2YW6HSHC5JGD4FHMFR6GRWP6WEZJSXX6JNQA, seg_pos={0 0}, upload=494748]: Done uploading piece. err: <nil> Dec 29 03:46:41.362228264 (564h17m41.233311299s) [noise=true , address=145.79.227.46:4609 , piece_id=P3GIG3OM7H432J3IGJIBJPSDGSRQ73MPIIWE5G4UYPFZHGHRIRMA, seg_pos={0 0}, upload=494748]: Done uploading piece. err: <nil> Dec 29 03:46:41.439067256 (564h17m41.310150301s) [noise=true , address=77.90.31.11:4581 , piece_id=STAAU2BS7L5JZ2E6HW65EKZ2LTJTHB2BQT3E5IQPZ6PMFTB5R2BA, seg_pos={0 0}, upload=494748]: Done uploading piece. err: <nil> Dec 29 03:46:41.493569326 (564h17m41.364652361s) [noise=true , address=45.91.116.4:4957 , piece_id=ZQ4K2ND3IIB4AML7CIDDZETG4RZ5L2RL7D2HJZMJ3WSGJZVE3BNQ, seg_pos={0 0}, upload=494748]: Done uploading piece. err: <nil> Dec 29 03:46:41.574947141 (564h17m41.446030176s) [noise=true , address=77.90.31.12:4604 , piece_id=FQDKYUMG2XHD4FQXGXZDTXG6MMTLNZYWDC47JJGUIOEXYTEYBMGA, seg_pos={0 0}, upload=494748]: Done uploading piece. err: <nil> Dec 29 03:46:41.704651464 (564h17m41.575734499s) [noise=true , address=77.90.31.16:4756 , piece_id=VH4O5TLJB2E2CI7RVAIB77AUU7T4Z7PQCFNHAHRHMWRK5U3SAU7A, seg_pos={0 0}, upload=494748]: Done uploading piece. err: <nil> Dec 29 03:46:42.010950742 (564h17m41.882033777s) [noise=true , address=77.90.31.14:4673 , piece_id=JQMMLQPK3NY4XLMX3G4DZUV7WEFNXDWGGJ4FH3BFC5YPHYWK5L4A, seg_pos={0 0}, upload=494748]: Done uploading piece. err: <nil> Dec 29 03:46:42.172005653 (564h17m42.043088688s) [noise=true , address=145.79.227.30:4779 , piece_id=FTSLUGXEJXOUE7MEFZWD6KJFAV6RGPWS7N7OJFSHADZZ7IIX55GQ, seg_pos={0 0}, upload=494748]: Done uploading piece. err: <nil> Dec 29 03:46:42.201208108 (564h17m42.072291153s) [noise=true , address=145.79.227.44:4208 , piece_id=ZMYK4NBXCN3LAVJHAEUFIGIPQRPQ4IFLCPKN52XDNPQ3CKGE77NQ, seg_pos={0 0}, upload=494748]: Done uploading piece. err: <nil> Dec 29 03:46:42.223177351 (564h17m42.094260387s) [noise=true , address=145.79.227.34:4925 , piece_id=XKSPDMDB2OUTOJVDJWJPCBBRLOHCL2NCKUBWK4IC43CWFC5LIJFA, seg_pos={0 0}, upload=494748]: Done uploading piece. err: <nil> Dec 29 03:46:42.276938741 (564h17m42.148021786s) [noise=true , address=77.90.31.18:4814 , piece_id=VUWCBNWEBOQP22BQVQ6M6KAHKOYJZNUIOB4ZYUZQDI7L4672NW7Q, seg_pos={0 0}, upload=494748]: Done uploading piece. err: <nil> Dec 29 03:46:42.367778062 (564h17m42.238861097s) [noise=true , address=77.90.31.13:4639 , piece_id=HMW2XHEVCPMULPKO7OUJYOU5BETJIXQVEXZSTY2KARXOD7HXT26Q, seg_pos={0 0}, upload=494748]: Done uploading piece. err: <nil> Dec 29 03:46:42.367802359 (564h17m42.238885394s) [seg_pos={0 0}, upload=494748]: Segment reached optimal threshold of 14 pieces. Dec 29 03:46:42.368052876 (564h17m42.239135931s) [noise=true , address=64.49.13.169:4710 , piece_id=Q27PBKCIHI36B5HIDKD6PKYVWTSI7QNV3CNEKVSSBKN3FVTPFDQA, seg_pos={0 0}, upload=494748]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hsbGK94RFwxic9F8NKp8kKvztegUhkNmYEor4QyLJmqF3seZf): protocol: write tcp 188.166.225.170:59868->64.49.13.169:4710: use of closed network connection; write tcp 188.166.225.170:59868->64.49.13.169:4710: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59868->64.49.13.169:4710: use of closed network connection Dec 29 03:46:42.368065499 (564h17m42.239148544s) [noise=true , address=64.49.13.169:4710 , piece_id=Q27PBKCIHI36B5HIDKD6PKYVWTSI7QNV3CNEKVSSBKN3FVTPFDQA, seg_pos={0 0}, upload=494748]: Long tail cancellation due to optimal threshold Dec 29 03:46:42.368081100 (564h17m42.239164135s) [noise=true , address=5.231.73.3:4023 , piece_id=OUGY6RHELOVD42KM3FMGBRXKRKGOUOHPY3DKSTJWGTUJ3Q5YROLQ, seg_pos={0 0}, upload=494748]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17xu6xC3uRdhHYHGDMzssSg81XyXWqFT6NbC44agM2Z3mf1Jt8): protocol: expected piece hash; context canceled; EOF Dec 29 03:46:42.368087922 (564h17m42.239170957s) [noise=true , address=5.231.73.3:4023 , piece_id=OUGY6RHELOVD42KM3FMGBRXKRKGOUOHPY3DKSTJWGTUJ3Q5YROLQ, seg_pos={0 0}, upload=494748]: Long tail cancellation due to optimal threshold Dec 29 03:46:42.368237757 (564h17m42.239320792s) [noise=true , address=145.79.227.40:4068 , piece_id=6RC5TFIDPFVPJXY4QNUX4GVDCR5562GP6J5IMVMWTQKUVUVC4XAQ, seg_pos={0 0}, upload=494748]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Lh9dYpHCxbs7gpZ4Xt7os5Gm5Nco6jrNW1V1FHddVUEMkFfPY): protocol: expected piece hash; context canceled; EOF Dec 29 03:46:42.368248137 (564h17m42.239331182s) [noise=true , address=145.79.227.40:4068 , piece_id=6RC5TFIDPFVPJXY4QNUX4GVDCR5562GP6J5IMVMWTQKUVUVC4XAQ, seg_pos={0 0}, upload=494748]: Long tail cancellation due to optimal threshold Dec 29 03:46:42.368433511 (564h17m42.239516546s) [noise=true , address=193.149.164.143:4699 , piece_id=BPV626GYTLB5KDERSZ5YMLKZ6MDJWOPK5MTXNJ4J4FU7PNHOCKIQ, seg_pos={0 0}, upload=494748]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1LncNSBhhreg9JTByJPSB7eGUEu88Eyd3jc9pg24VsEjp4FULq): protocol: expected piece hash; context canceled; EOF Dec 29 03:46:42.368442537 (564h17m42.239525573s) [noise=true , address=193.149.164.143:4699 , piece_id=BPV626GYTLB5KDERSZ5YMLKZ6MDJWOPK5MTXNJ4J4FU7PNHOCKIQ, seg_pos={0 0}, upload=494748]: Long tail cancellation due to optimal threshold Dec 29 03:46:42.368532690 (564h17m42.239615725s) [seg_pos={0 0}, upload=494748]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 03:46:42.368616819 (564h17m42.239699854s) [upload=494748]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 03:46:42.368621358 (564h17m42.239704393s) [upload=494748]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 03:46:42.667817119 (564h17m42.538900164s) [upload=494748]: Done uploading segments... Dec 29 03:46:42.667836256 (564h17m42.538919301s) [upload=494748]: Upload finished. err: <nil> Duration: 53.280057391s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 7 out of 60 is failed (threshold=6) [....................XX................X......XXX......X.....] duration: 3.541463721s trace: https://jaeger.dev.storj.io/trace/f91d358bd5ae577 FAIL (2025-12-29 03:50:23 UTC) orbiter warning: context deadline exceeded Duration: 3.541488538s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 33 out of 60 is failed (threshold=3) [X.X.....X.X..XXXX..XXX.XX...X.X.....XXX...XXXXXXX.XX.XXX.XXX] duration: 30.002138532s trace: https://jaeger.dev.storj.io/trace/219b97baeff818e0 FAIL (2025-12-29 03:50:20 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks, Piece ID: USLJBKLNFPRVQANBT6J32IYVX4X4REWOMK3VRDNODPERPSOPMCIA) context deadline exceeded; piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: 4DBGD3THLBYYRTCVSYPJ5BYGFO2RH2YN5266H2474NLNDOSLPG5Q) context deadline exceeded; piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: 46WJWPG3YJWV657VZ553TBT5WJ7R7M75D3GWVUMZH474DITEXXRA) context deadline exceeded ======LOGS====== 1786125 (564h20m51.51286917s) [upload=494789]: Done starting upload Dec 29 03:49:51.641866909 (564h20m51.512949954s) [upload=494789]: Uploading segments... Dec 29 03:49:51.642009370 (564h20m51.513092405s) [upload=494789]: Got next segment. Inline: false Dec 29 03:49:51.642043364 (564h20m51.5131264s) [upload=494789]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 03:49:51.642047853 (564h20m51.513130898s) [upload=494789]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 03:49:51.887145146 (564h20m51.758228191s) [seg_pos={0 0}, upload=494789]: Begin upload segment... Dec 29 03:49:51.887203567 (564h20m51.758286612s) [seg_pos={0 0}, upload=494789]: Done begin upload segment. Dec 29 03:49:51.887241549 (564h20m51.758324594s) [noise=true , address=145.79.227.46:4602 , piece_id=HDYP3DLHTHQXF6HLQNPEYJAC6BMORVTVFE6OH7S2SCUDIIV62D7A, seg_pos={0 0}, upload=494789]: Uploading piece... Dec 29 03:49:51.887247430 (564h20m51.758330475s) [noise=true , address=145.79.227.26:4653 , piece_id=D22BYB2TLMEPUMTCKVC2D4WXIMZY72FSNED57APJAAVJJTABDFAQ, seg_pos={0 0}, upload=494789]: Uploading piece... Dec 29 03:49:51.888105163 (564h20m51.759188218s) [noise=true , address=45.91.116.11:4117 , piece_id=4DBGD3THLBYYRTCVSYPJ5BYGFO2RH2YN5266H2474NLNDOSLPG5Q, seg_pos={0 0}, upload=494789]: Uploading piece... Dec 29 03:49:51.888426826 (564h20m51.759509861s) [noise=true , address=145.79.227.12:4767 , piece_id=O7ISLSMMMK4JLRHDVJQL6MP3MHB4IW3WXZBSR2XXWE7ERFREN3PA, seg_pos={0 0}, upload=494789]: Uploading piece... Dec 29 03:49:51.888685338 (564h20m51.759768373s) [noise=true , address=145.79.227.15:4875 , piece_id=YP27ZOVPGLMTFSWBO3QLQQCAUXIDCZHEXFGD6EEJCDY5VGE4CGVA, seg_pos={0 0}, upload=494789]: Uploading piece... Dec 29 03:49:51.889024915 (564h20m51.76010796s) [noise=true , address=77.90.31.12:4590 , piece_id=ONQ7S6ATSH5OAUDCOJVIAGFVTNPMZNYDYZ6Z7ECP5I7C3L3BKESA, seg_pos={0 0}, upload=494789]: Uploading piece... Dec 29 03:49:51.889340917 (564h20m51.760423962s) [noise=true , address=45.91.116.8:4045 , piece_id=46WJWPG3YJWV657VZ553TBT5WJ7R7M75D3GWVUMZH474DITEXXRA, seg_pos={0 0}, upload=494789]: Uploading piece... Dec 29 03:49:51.889667558 (564h20m51.760750613s) [noise=true , address=45.91.116.7:4012 , piece_id=6P3PM6EXKO5VDOHZVTJ4NIF2OO6CZMIUOVHYYFLQWJ2GNALNZRGA, seg_pos={0 0}, upload=494789]: Uploading piece... Dec 29 03:49:51.890043454 (564h20m51.761126499s) [noise=true , address=45.91.116.12:4226 , piece_id=PHDVWJFB6KY76HXLJEPWSZ673BD4IMJCYFI7PRC3ESQHWFF6XO5A, seg_pos={0 0}, upload=494789]: Uploading piece... Dec 29 03:49:51.890425321 (564h20m51.761508356s) [noise=true , address=77.90.31.11:4556 , piece_id=USLJBKLNFPRVQANBT6J32IYVX4X4REWOMK3VRDNODPERPSOPMCIA, seg_pos={0 0}, upload=494789]: Uploading piece... Dec 29 03:49:51.890893684 (564h20m51.761976729s) [noise=true , address=145.79.227.7:4294 , piece_id=XOTCZXSGA32R6C2B6IIF6VTGORCMZ44HQA5RSD3TEVA7UP664YNA, seg_pos={0 0}, upload=494789]: Uploading piece... Dec 29 03:49:51.891299095 (564h20m51.76238214s) [noise=true , address=145.79.227.51:4410 , piece_id=Y2V2C4ENRMSORX2FZ3GUWQFFLBXZGZ24WE5SCTKTXWPZM3IL3BLA, seg_pos={0 0}, upload=494789]: Uploading piece... Dec 29 03:49:51.891679530 (564h20m51.762762575s) [noise=true , address=145.79.227.14:4840 , piece_id=UX6PQ7SLFK2QJJT5TBPZ52K74NFA77NN7HOBYVEND77FSPGZ3DOQ, seg_pos={0 0}, upload=494789]: Uploading piece... Dec 29 03:49:51.891977127 (564h20m51.763060162s) [noise=true , address=145.79.227.24:4582 , piece_id=5BHCRAGHR3D6CBYHEHTSIAGLHODZHDBYYSBPSDSFNZHRML7BDWPA, seg_pos={0 0}, upload=494789]: Uploading piece... Dec 29 03:49:51.892347311 (564h20m51.763430356s) [noise=true , address=45.91.116.13:4155 , piece_id=I4ZG6EWYL7VVFM6OWF4R2CVNDYHJYPWOB4OEXLSCNTLVD5OZ6HPA, seg_pos={0 0}, upload=494789]: Uploading piece... Dec 29 03:49:51.892668313 (564h20m51.763751358s) [noise=true , address=145.79.227.19:4079 , piece_id=K3MEZZGFDGDQJKR2X2NGNOY7LNT3PYQFEA3XHGNY4WL7U3U2ETCQ, seg_pos={0 0}, upload=494789]: Uploading piece... Dec 29 03:49:53.593529499 (564h20m53.464612534s) [upload=494789]: Next returned nil segment Dec 29 03:49:53.593552994 (564h20m53.464636039s) [upload=494789]: Waiting for error group managing segments... Dec 29 03:49:53.871078986 (564h20m53.742162031s) [noise=true , address=145.79.227.15:4875 , piece_id=YP27ZOVPGLMTFSWBO3QLQQCAUXIDCZHEXFGD6EEJCDY5VGE4CGVA, seg_pos={0 0}, upload=494789]: Done uploading piece. err: <nil> Dec 29 03:49:53.872519960 (564h20m53.743602995s) [noise=true , address=145.79.227.14:4840 , piece_id=UX6PQ7SLFK2QJJT5TBPZ52K74NFA77NN7HOBYVEND77FSPGZ3DOQ, seg_pos={0 0}, upload=494789]: Done uploading piece. err: <nil> Dec 29 03:49:53.877316472 (564h20m53.748399517s) [noise=true , address=145.79.227.26:4653 , piece_id=D22BYB2TLMEPUMTCKVC2D4WXIMZY72FSNED57APJAAVJJTABDFAQ, seg_pos={0 0}, upload=494789]: Done uploading piece. err: <nil> Dec 29 03:49:53.877943296 (564h20m53.749026361s) [noise=true , address=145.79.227.46:4602 , piece_id=HDYP3DLHTHQXF6HLQNPEYJAC6BMORVTVFE6OH7S2SCUDIIV62D7A, seg_pos={0 0}, upload=494789]: Done uploading piece. err: <nil> Dec 29 03:49:53.882299011 (564h20m53.753382046s) [noise=true , address=145.79.227.12:4767 , piece_id=O7ISLSMMMK4JLRHDVJQL6MP3MHB4IW3WXZBSR2XXWE7ERFREN3PA, seg_pos={0 0}, upload=494789]: Done uploading piece. err: <nil> Dec 29 03:49:53.894460667 (564h20m53.765543712s) [noise=true , address=145.79.227.19:4079 , piece_id=K3MEZZGFDGDQJKR2X2NGNOY7LNT3PYQFEA3XHGNY4WL7U3U2ETCQ, seg_pos={0 0}, upload=494789]: Done uploading piece. err: <nil> Dec 29 03:49:53.898466195 (564h20m53.76954923s) [noise=true , address=145.79.227.7:4294 , piece_id=XOTCZXSGA32R6C2B6IIF6VTGORCMZ44HQA5RSD3TEVA7UP664YNA, seg_pos={0 0}, upload=494789]: Done uploading piece. err: <nil> Dec 29 03:49:53.901614610 (564h20m53.772697645s) [noise=true , address=145.79.227.51:4410 , piece_id=Y2V2C4ENRMSORX2FZ3GUWQFFLBXZGZ24WE5SCTKTXWPZM3IL3BLA, seg_pos={0 0}, upload=494789]: Done uploading piece. err: <nil> Dec 29 03:49:53.902117547 (564h20m53.773200592s) [noise=true , address=145.79.227.24:4582 , piece_id=5BHCRAGHR3D6CBYHEHTSIAGLHODZHDBYYSBPSDSFNZHRML7BDWPA, seg_pos={0 0}, upload=494789]: Done uploading piece. err: <nil> Dec 29 03:49:54.124803635 (564h20m53.99588667s) [noise=true , address=45.91.116.8:4045 , piece_id=46WJWPG3YJWV657VZ553TBT5WJ7R7M75D3GWVUMZH474DITEXXRA, seg_pos={0 0}, upload=494789]: Done uploading piece. err: <nil> Dec 29 03:49:54.131497121 (564h20m54.002580166s) [noise=true , address=45.91.116.13:4155 , piece_id=I4ZG6EWYL7VVFM6OWF4R2CVNDYHJYPWOB4OEXLSCNTLVD5OZ6HPA, seg_pos={0 0}, upload=494789]: Done uploading piece. err: <nil> Dec 29 03:49:54.134081162 (564h20m54.005164197s) [noise=true , address=77.90.31.11:4556 , piece_id=USLJBKLNFPRVQANBT6J32IYVX4X4REWOMK3VRDNODPERPSOPMCIA, seg_pos={0 0}, upload=494789]: Done uploading piece. err: <nil> Dec 29 03:49:54.245202172 (564h20m54.116285207s) [noise=true , address=45.91.116.11:4117 , piece_id=4DBGD3THLBYYRTCVSYPJ5BYGFO2RH2YN5266H2474NLNDOSLPG5Q, seg_pos={0 0}, upload=494789]: Done uploading piece. err: <nil> Dec 29 03:49:54.245222681 (564h20m54.116305717s) [seg_pos={0 0}, upload=494789]: Segment reached optimal threshold of 13 pieces. Dec 29 03:49:54.245385933 (564h20m54.116468968s) [noise=true , address=45.91.116.12:4226 , piece_id=PHDVWJFB6KY76HXLJEPWSZ673BD4IMJCYFI7PRC3ESQHWFF6XO5A, seg_pos={0 0}, upload=494789]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Dec 29 03:49:54.245392585 (564h20m54.11647562s) [noise=true , address=45.91.116.12:4226 , piece_id=PHDVWJFB6KY76HXLJEPWSZ673BD4IMJCYFI7PRC3ESQHWFF6XO5A, seg_pos={0 0}, upload=494789]: Long tail cancellation due to optimal threshold Dec 29 03:49:54.245690281 (564h20m54.116773326s) [noise=true , address=45.91.116.7:4012 , piece_id=6P3PM6EXKO5VDOHZVTJ4NIF2OO6CZMIUOVHYYFLQWJ2GNALNZRGA, seg_pos={0 0}, upload=494789]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t): protocol: write tcp 188.166.225.170:58222->45.91.116.7:4012: use of closed network connection; write tcp 188.166.225.170:58222->45.91.116.7:4012: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58222->45.91.116.7:4012: use of closed network connection Dec 29 03:49:54.245706833 (564h20m54.116789869s) [noise=true , address=45.91.116.7:4012 , piece_id=6P3PM6EXKO5VDOHZVTJ4NIF2OO6CZMIUOVHYYFLQWJ2GNALNZRGA, seg_pos={0 0}, upload=494789]: Long tail cancellation due to optimal threshold Dec 29 03:49:54.245731541 (564h20m54.116814576s) [noise=true , address=77.90.31.12:4590 , piece_id=ONQ7S6ATSH5OAUDCOJVIAGFVTNPMZNYDYZ6Z7ECP5I7C3L3BKESA, seg_pos={0 0}, upload=494789]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: read tcp 188.166.225.170:37098->77.90.31.12:4590: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 29 03:49:54.245739345 (564h20m54.11682238s) [noise=true , address=77.90.31.12:4590 , piece_id=ONQ7S6ATSH5OAUDCOJVIAGFVTNPMZNYDYZ6Z7ECP5I7C3L3BKESA, seg_pos={0 0}, upload=494789]: Long tail cancellation due to optimal threshold Dec 29 03:49:54.245754684 (564h20m54.116837719s) [seg_pos={0 0}, upload=494789]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 03:49:54.245854745 (564h20m54.11693779s) [upload=494789]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 03:49:54.245860155 (564h20m54.11694319s) [upload=494789]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 03:49:54.530152570 (564h20m54.401235625s) [upload=494789]: Done uploading segments... Dec 29 03:49:54.530178379 (564h20m54.401261415s) [upload=494789]: Upload finished. err: <nil> Duration: 30.00215807s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 7 out of 60 is failed (threshold=6) [.....................XX................X......XXX......X....] duration: 5.001341247s trace: https://jaeger.dev.storj.io/trace/f91d358bd5ae577 FAIL (2025-12-29 03:49:59 UTC) orbiter warning: context deadline exceeded Duration: 5.001368409s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [..........................X...................X.......X.X...] duration: 46.314244628s trace: https://jaeger.dev.storj.io/trace/262a3b4a3a5f2b2e FAIL (2025-12-29 03:49:42 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== se=true , address=64.49.13.27:4958 , piece_id=RNO7Q6EBDZOFRIDD6M4BU53OXVB52P3ZQZKC7ZQ6VEOD2AIO6MEQ, seg_pos={0 0}, upload=494748]: Uploading piece... Dec 29 03:46:39.389590023 (564h17m39.260673058s) [noise=true , address=77.90.31.11:4581 , piece_id=STAAU2BS7L5JZ2E6HW65EKZ2LTJTHB2BQT3E5IQPZ6PMFTB5R2BA, seg_pos={0 0}, upload=494748]: Uploading piece... Dec 29 03:46:39.389892929 (564h17m39.260975964s) [noise=true , address=77.90.31.16:4756 , piece_id=VH4O5TLJB2E2CI7RVAIB77AUU7T4Z7PQCFNHAHRHMWRK5U3SAU7A, seg_pos={0 0}, upload=494748]: Uploading piece... Dec 29 03:46:39.390155760 (564h17m39.261238805s) [noise=true , address=5.231.73.3:4023 , piece_id=OUGY6RHELOVD42KM3FMGBRXKRKGOUOHPY3DKSTJWGTUJ3Q5YROLQ, seg_pos={0 0}, upload=494748]: Uploading piece... Dec 29 03:46:39.390429110 (564h17m39.261512145s) [noise=true , address=145.79.227.44:4208 , piece_id=ZMYK4NBXCN3LAVJHAEUFIGIPQRPQ4IFLCPKN52XDNPQ3CKGE77NQ, seg_pos={0 0}, upload=494748]: Uploading piece... Dec 29 03:46:39.390774769 (564h17m39.261857804s) [noise=true , address=64.49.13.169:4710 , piece_id=Q27PBKCIHI36B5HIDKD6PKYVWTSI7QNV3CNEKVSSBKN3FVTPFDQA, seg_pos={0 0}, upload=494748]: Uploading piece... Dec 29 03:46:39.391067896 (564h17m39.262150931s) [noise=true , address=145.79.227.40:4068 , piece_id=6RC5TFIDPFVPJXY4QNUX4GVDCR5562GP6J5IMVMWTQKUVUVC4XAQ, seg_pos={0 0}, upload=494748]: Uploading piece... Dec 29 03:46:39.391349352 (564h17m39.262432397s) [noise=true , address=77.90.31.14:4673 , piece_id=JQMMLQPK3NY4XLMX3G4DZUV7WEFNXDWGGJ4FH3BFC5YPHYWK5L4A, seg_pos={0 0}, upload=494748]: Uploading piece... Dec 29 03:46:39.391620058 (564h17m39.262703113s) [noise=true , address=145.79.227.30:4779 , piece_id=FTSLUGXEJXOUE7MEFZWD6KJFAV6RGPWS7N7OJFSHADZZ7IIX55GQ, seg_pos={0 0}, upload=494748]: Uploading piece... Dec 29 03:46:39.391987468 (564h17m39.263070503s) [noise=true , address=77.90.31.13:4639 , piece_id=HMW2XHEVCPMULPKO7OUJYOU5BETJIXQVEXZSTY2KARXOD7HXT26Q, seg_pos={0 0}, upload=494748]: Uploading piece... Dec 29 03:46:39.392352663 (564h17m39.263435698s) [noise=true , address=77.90.31.18:4814 , piece_id=VUWCBNWEBOQP22BQVQ6M6KAHKOYJZNUIOB4ZYUZQDI7L4672NW7Q, seg_pos={0 0}, upload=494748]: Uploading piece... Dec 29 03:46:39.393422620 (564h17m39.264505656s) [noise=true , address=77.90.31.12:4604 , piece_id=FQDKYUMG2XHD4FQXGXZDTXG6MMTLNZYWDC47JJGUIOEXYTEYBMGA, seg_pos={0 0}, upload=494748]: Uploading piece... Dec 29 03:46:39.394657942 (564h17m39.265740977s) [noise=true , address=145.79.227.25:4652 , piece_id=S2HK4O5A2SDILSDE2YW6HSHC5JGD4FHMFR6GRWP6WEZJSXX6JNQA, seg_pos={0 0}, upload=494748]: Uploading piece... Dec 29 03:46:39.395699916 (564h17m39.266782951s) [noise=true , address=145.79.227.34:4925 , piece_id=XKSPDMDB2OUTOJVDJWJPCBBRLOHCL2NCKUBWK4IC43CWFC5LIJFA, seg_pos={0 0}, upload=494748]: Uploading piece... Dec 29 03:46:39.396753092 (564h17m39.267836127s) [noise=true , address=193.149.164.143:4699 , piece_id=BPV626GYTLB5KDERSZ5YMLKZ6MDJWOPK5MTXNJ4J4FU7PNHOCKIQ, seg_pos={0 0}, upload=494748]: Uploading piece... Dec 29 03:46:39.398134963 (564h17m39.269217998s) [noise=true , address=145.79.227.12:4773 , piece_id=L2O453KQ6JWGDEJT2K4VA4M2MVBSHKDJKNE7UASDJ5KPZNH4L2HA, seg_pos={0 0}, upload=494748]: Uploading piece... Dec 29 03:46:40.716827029 (564h17m40.587910064s) [upload=494748]: Next returned nil segment Dec 29 03:46:40.716839693 (564h17m40.587922728s) [upload=494748]: Waiting for error group managing segments... Dec 29 03:46:41.051033808 (564h17m40.922116843s) [noise=true , address=64.49.13.27:4958 , piece_id=RNO7Q6EBDZOFRIDD6M4BU53OXVB52P3ZQZKC7ZQ6VEOD2AIO6MEQ, seg_pos={0 0}, upload=494748]: Done uploading piece. err: <nil> Dec 29 03:46:41.262360402 (564h17m41.133443447s) [noise=true , address=145.79.227.12:4773 , piece_id=L2O453KQ6JWGDEJT2K4VA4M2MVBSHKDJKNE7UASDJ5KPZNH4L2HA, seg_pos={0 0}, upload=494748]: Done uploading piece. err: <nil> Dec 29 03:46:41.265241008 (564h17m41.136324063s) [noise=true , address=145.79.227.25:4652 , piece_id=S2HK4O5A2SDILSDE2YW6HSHC5JGD4FHMFR6GRWP6WEZJSXX6JNQA, seg_pos={0 0}, upload=494748]: Done uploading piece. err: <nil> Dec 29 03:46:41.362228264 (564h17m41.233311299s) [noise=true , address=145.79.227.46:4609 , piece_id=P3GIG3OM7H432J3IGJIBJPSDGSRQ73MPIIWE5G4UYPFZHGHRIRMA, seg_pos={0 0}, upload=494748]: Done uploading piece. err: <nil> Dec 29 03:46:41.439067256 (564h17m41.310150301s) [noise=true , address=77.90.31.11:4581 , piece_id=STAAU2BS7L5JZ2E6HW65EKZ2LTJTHB2BQT3E5IQPZ6PMFTB5R2BA, seg_pos={0 0}, upload=494748]: Done uploading piece. err: <nil> Dec 29 03:46:41.493569326 (564h17m41.364652361s) [noise=true , address=45.91.116.4:4957 , piece_id=ZQ4K2ND3IIB4AML7CIDDZETG4RZ5L2RL7D2HJZMJ3WSGJZVE3BNQ, seg_pos={0 0}, upload=494748]: Done uploading piece. err: <nil> Dec 29 03:46:41.574947141 (564h17m41.446030176s) [noise=true , address=77.90.31.12:4604 , piece_id=FQDKYUMG2XHD4FQXGXZDTXG6MMTLNZYWDC47JJGUIOEXYTEYBMGA, seg_pos={0 0}, upload=494748]: Done uploading piece. err: <nil> Dec 29 03:46:41.704651464 (564h17m41.575734499s) [noise=true , address=77.90.31.16:4756 , piece_id=VH4O5TLJB2E2CI7RVAIB77AUU7T4Z7PQCFNHAHRHMWRK5U3SAU7A, seg_pos={0 0}, upload=494748]: Done uploading piece. err: <nil> Dec 29 03:46:42.010950742 (564h17m41.882033777s) [noise=true , address=77.90.31.14:4673 , piece_id=JQMMLQPK3NY4XLMX3G4DZUV7WEFNXDWGGJ4FH3BFC5YPHYWK5L4A, seg_pos={0 0}, upload=494748]: Done uploading piece. err: <nil> Dec 29 03:46:42.172005653 (564h17m42.043088688s) [noise=true , address=145.79.227.30:4779 , piece_id=FTSLUGXEJXOUE7MEFZWD6KJFAV6RGPWS7N7OJFSHADZZ7IIX55GQ, seg_pos={0 0}, upload=494748]: Done uploading piece. err: <nil> Dec 29 03:46:42.201208108 (564h17m42.072291153s) [noise=true , address=145.79.227.44:4208 , piece_id=ZMYK4NBXCN3LAVJHAEUFIGIPQRPQ4IFLCPKN52XDNPQ3CKGE77NQ, seg_pos={0 0}, upload=494748]: Done uploading piece. err: <nil> Dec 29 03:46:42.223177351 (564h17m42.094260387s) [noise=true , address=145.79.227.34:4925 , piece_id=XKSPDMDB2OUTOJVDJWJPCBBRLOHCL2NCKUBWK4IC43CWFC5LIJFA, seg_pos={0 0}, upload=494748]: Done uploading piece. err: <nil> Dec 29 03:46:42.276938741 (564h17m42.148021786s) [noise=true , address=77.90.31.18:4814 , piece_id=VUWCBNWEBOQP22BQVQ6M6KAHKOYJZNUIOB4ZYUZQDI7L4672NW7Q, seg_pos={0 0}, upload=494748]: Done uploading piece. err: <nil> Dec 29 03:46:42.367778062 (564h17m42.238861097s) [noise=true , address=77.90.31.13:4639 , piece_id=HMW2XHEVCPMULPKO7OUJYOU5BETJIXQVEXZSTY2KARXOD7HXT26Q, seg_pos={0 0}, upload=494748]: Done uploading piece. err: <nil> Dec 29 03:46:42.367802359 (564h17m42.238885394s) [seg_pos={0 0}, upload=494748]: Segment reached optimal threshold of 14 pieces. Dec 29 03:46:42.368052876 (564h17m42.239135931s) [noise=true , address=64.49.13.169:4710 , piece_id=Q27PBKCIHI36B5HIDKD6PKYVWTSI7QNV3CNEKVSSBKN3FVTPFDQA, seg_pos={0 0}, upload=494748]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hsbGK94RFwxic9F8NKp8kKvztegUhkNmYEor4QyLJmqF3seZf): protocol: write tcp 188.166.225.170:59868->64.49.13.169:4710: use of closed network connection; write tcp 188.166.225.170:59868->64.49.13.169:4710: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59868->64.49.13.169:4710: use of closed network connection Dec 29 03:46:42.368065499 (564h17m42.239148544s) [noise=true , address=64.49.13.169:4710 , piece_id=Q27PBKCIHI36B5HIDKD6PKYVWTSI7QNV3CNEKVSSBKN3FVTPFDQA, seg_pos={0 0}, upload=494748]: Long tail cancellation due to optimal threshold Dec 29 03:46:42.368081100 (564h17m42.239164135s) [noise=true , address=5.231.73.3:4023 , piece_id=OUGY6RHELOVD42KM3FMGBRXKRKGOUOHPY3DKSTJWGTUJ3Q5YROLQ, seg_pos={0 0}, upload=494748]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17xu6xC3uRdhHYHGDMzssSg81XyXWqFT6NbC44agM2Z3mf1Jt8): protocol: expected piece hash; context canceled; EOF Dec 29 03:46:42.368087922 (564h17m42.239170957s) [noise=true , address=5.231.73.3:4023 , piece_id=OUGY6RHELOVD42KM3FMGBRXKRKGOUOHPY3DKSTJWGTUJ3Q5YROLQ, seg_pos={0 0}, upload=494748]: Long tail cancellation due to optimal threshold Dec 29 03:46:42.368237757 (564h17m42.239320792s) [noise=true , address=145.79.227.40:4068 , piece_id=6RC5TFIDPFVPJXY4QNUX4GVDCR5562GP6J5IMVMWTQKUVUVC4XAQ, seg_pos={0 0}, upload=494748]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Lh9dYpHCxbs7gpZ4Xt7os5Gm5Nco6jrNW1V1FHddVUEMkFfPY): protocol: expected piece hash; context canceled; EOF Dec 29 03:46:42.368248137 (564h17m42.239331182s) [noise=true , address=145.79.227.40:4068 , piece_id=6RC5TFIDPFVPJXY4QNUX4GVDCR5562GP6J5IMVMWTQKUVUVC4XAQ, seg_pos={0 0}, upload=494748]: Long tail cancellation due to optimal threshold Dec 29 03:46:42.368433511 (564h17m42.239516546s) [noise=true , address=193.149.164.143:4699 , piece_id=BPV626GYTLB5KDERSZ5YMLKZ6MDJWOPK5MTXNJ4J4FU7PNHOCKIQ, seg_pos={0 0}, upload=494748]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1LncNSBhhreg9JTByJPSB7eGUEu88Eyd3jc9pg24VsEjp4FULq): protocol: expected piece hash; context canceled; EOF Dec 29 03:46:42.368442537 (564h17m42.239525573s) [noise=true , address=193.149.164.143:4699 , piece_id=BPV626GYTLB5KDERSZ5YMLKZ6MDJWOPK5MTXNJ4J4FU7PNHOCKIQ, seg_pos={0 0}, upload=494748]: Long tail cancellation due to optimal threshold Dec 29 03:46:42.368532690 (564h17m42.239615725s) [seg_pos={0 0}, upload=494748]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 03:46:42.368616819 (564h17m42.239699854s) [upload=494748]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 03:46:42.368621358 (564h17m42.239704393s) [upload=494748]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 03:46:42.667817119 (564h17m42.538900164s) [upload=494748]: Done uploading segments... Dec 29 03:46:42.667836256 (564h17m42.538919301s) [upload=494748]: Upload finished. err: <nil> Duration: 46.314258755s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 33 out of 60 is failed (threshold=3) [XX.X.....X.X..XXXX..XXX.XX...X.X.....XXX...XXXXXXX.XX.XXX.XX] duration: 30.001880853s trace: https://jaeger.dev.storj.io/trace/43e312818d1dd217 FAIL (2025-12-29 03:49:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, Piece ID: KSYB52NEH536LENGKSTDEQ2D2OAIV5W55JMPIMOZITS5Z3KNGMGQ) context deadline exceeded; piecestore: (Node ID: 12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, Piece ID: QGXA7QV3CSHUF2T5SGM2VFUYCMACQJ6HYMZCAXNFV5L2UZIYY6PA) context deadline exceeded ======LOGS====== Dec 29 03:48:53.955668368 (564h19m53.826751403s) [upload=494778]: Starting upload Dec 29 03:48:53.955723434 (564h19m53.826806469s) [upload=494778]: Done starting upload Dec 29 03:48:53.955917593 (564h19m53.827000628s) [upload=494778]: Uploading segments... Dec 29 03:48:53.955981104 (564h19m53.827064149s) [upload=494778]: Got next segment. Inline: false Dec 29 03:48:53.956008466 (564h19m53.827091511s) [upload=494778]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 03:48:53.956013045 (564h19m53.82709608s) [upload=494778]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 03:48:54.202013516 (564h19m54.073096551s) [seg_pos={0 0}, upload=494778]: Begin upload segment... Dec 29 03:48:54.202095613 (564h19m54.073178648s) [seg_pos={0 0}, upload=494778]: Done begin upload segment. Dec 29 03:48:54.202143314 (564h19m54.073226349s) [noise=true , address=145.79.227.46:4601 , piece_id=333P6ZRMBYPS3P4EIOXO5XNRSFHR7FEQZ72664KAOXWTLN7LIFGA, seg_pos={0 0}, upload=494778]: Uploading piece... Dec 29 03:48:54.202168783 (564h19m54.073251828s) [noise=true , address=145.79.227.48:4303 , piece_id=ABRJ2AZBSJMAZUBM2VQU5IOXWOKDOTRL4SWTI22O6OVZMU7YKFSQ, seg_pos={0 0}, upload=494778]: Uploading piece... Dec 29 03:48:54.202786930 (564h19m54.073869965s) [noise=true , address=145.79.227.12:4765 , piece_id=I4FK33CTEWSVLSYMQ5SLPBH5BJGDWSNBT7XIGTNLKDCYNUO2NGSQ, seg_pos={0 0}, upload=494778]: Uploading piece... Dec 29 03:48:54.203085979 (564h19m54.074169014s) [noise=true , address=45.91.116.6:4975 , piece_id=VPNQOMA3B7CZH66SBDADW3KLXNS7J5CV2AV4OJ5ZXQZXV4R5CFQQ, seg_pos={0 0}, upload=494778]: Uploading piece... Dec 29 03:48:54.203380751 (564h19m54.074463786s) [noise=true , address=145.79.227.19:4077 , piece_id=FEEK7VYRGHQXCTM4OL5N25LP4IF4BR24UPX63NGX4HB2HTFRNOTQ, seg_pos={0 0}, upload=494778]: Uploading piece... Dec 29 03:48:54.203759071 (564h19m54.074842106s) [noise=true , address=45.91.116.5:4298 , piece_id=A47N3HMV757ODUKB4WDQL4QRIVESX6UOC4S3TPLKCPWZQRWBDUBA, seg_pos={0 0}, upload=494778]: Uploading piece... Dec 29 03:48:54.204919330 (564h19m54.076002375s) [noise=true , address=45.91.116.14:4192 , piece_id=EIF6RYZB3K6F5EGOR46V2ZOKPAFLMEYG5GIESTWHX4RT4O5L77GQ, seg_pos={0 0}, upload=494778]: Uploading piece... Dec 29 03:48:54.206041366 (564h19m54.077124401s) [noise=true , address=45.91.116.12:4225 , piece_id=QGXA7QV3CSHUF2T5SGM2VFUYCMACQJ6HYMZCAXNFV5L2UZIYY6PA, seg_pos={0 0}, upload=494778]: Uploading piece... Dec 29 03:48:54.207379644 (564h19m54.078462679s) [noise=true , address=77.90.31.10:4519 , piece_id=HBIGUFSCUTHA7IENQMDZRFUEQTSFZZNNSBG4FD2NHBXBVR722PFA, seg_pos={0 0}, upload=494778]: Uploading piece... Dec 29 03:48:54.208491391 (564h19m54.079574436s) [noise=true , address=145.79.227.14:4837 , piece_id=G4RETDUB5HMZKSPF5XK63PG6N2ESPEKY4PP7TVNDUMXKVYKPXTZQ, seg_pos={0 0}, upload=494778]: Uploading piece... Dec 29 03:48:54.209564254 (564h19m54.0806473s) [noise=true , address=145.79.227.17:4972 , piece_id=SBTRXANYZ6RSIWDWM7GAKYLYQYAOTJENUKG26D4XMDYDEVGVWPCQ, seg_pos={0 0}, upload=494778]: Uploading piece... Dec 29 03:48:54.210817391 (564h19m54.081900426s) [noise=true , address=45.91.116.10:4084 , piece_id=32I5AOGPLGKG7KTWTXY2YPEUGDYWPOZKQMZM7XTP3S24VTUKQ5ZA, seg_pos={0 0}, upload=494778]: Uploading piece... Dec 29 03:48:54.212116314 (564h19m54.083199349s) [noise=true , address=145.79.227.37:4566 , piece_id=IJ44CIMIJLNCIISU3UNOAUTSBWRCDU6Y57UZ5Q7M72IWCKX734LA, seg_pos={0 0}, upload=494778]: Uploading piece... Dec 29 03:48:54.212751713 (564h19m54.083834748s) [noise=true , address=77.90.31.21:4878 , piece_id=KSYB52NEH536LENGKSTDEQ2D2OAIV5W55JMPIMOZITS5Z3KNGMGQ, seg_pos={0 0}, upload=494778]: Uploading piece... Dec 29 03:48:54.214099460 (564h19m54.085182505s) [noise=true , address=145.79.227.49:4337 , piece_id=VWXWGTHCI7NWWB237ZGJDOCMHMHZV6DV7YMIJ2LRPH4DAZ5Y6WWA, seg_pos={0 0}, upload=494778]: Uploading piece... Dec 29 03:48:54.214645771 (564h19m54.085728806s) [noise=true , address=145.79.227.26:4655 , piece_id=4NU3DDPWAXNSZHQKGDC3UA2IS5OEI5IBHSJNLYYEEFAQH26SRF5Q, seg_pos={0 0}, upload=494778]: Uploading piece... Dec 29 03:48:55.900673039 (564h19m55.771756085s) [upload=494778]: Next returned nil segment Dec 29 03:48:55.900711162 (564h19m55.771794197s) [upload=494778]: Waiting for error group managing segments... Dec 29 03:48:56.184406872 (564h19m56.055489907s) [noise=true , address=145.79.227.48:4303 , piece_id=ABRJ2AZBSJMAZUBM2VQU5IOXWOKDOTRL4SWTI22O6OVZMU7YKFSQ, seg_pos={0 0}, upload=494778]: Done uploading piece. err: <nil> Dec 29 03:48:56.190641123 (564h19m56.061724158s) [noise=true , address=145.79.227.17:4972 , piece_id=SBTRXANYZ6RSIWDWM7GAKYLYQYAOTJENUKG26D4XMDYDEVGVWPCQ, seg_pos={0 0}, upload=494778]: Done uploading piece. err: <nil> Dec 29 03:48:56.192675556 (564h19m56.063758591s) [noise=true , address=145.79.227.14:4837 , piece_id=G4RETDUB5HMZKSPF5XK63PG6N2ESPEKY4PP7TVNDUMXKVYKPXTZQ, seg_pos={0 0}, upload=494778]: Done uploading piece. err: <nil> Dec 29 03:48:56.200407780 (564h19m56.071490815s) [noise=true , address=145.79.227.19:4077 , piece_id=FEEK7VYRGHQXCTM4OL5N25LP4IF4BR24UPX63NGX4HB2HTFRNOTQ, seg_pos={0 0}, upload=494778]: Done uploading piece. err: <nil> Dec 29 03:48:56.204901707 (564h19m56.075984752s) [noise=true , address=145.79.227.12:4765 , piece_id=I4FK33CTEWSVLSYMQ5SLPBH5BJGDWSNBT7XIGTNLKDCYNUO2NGSQ, seg_pos={0 0}, upload=494778]: Done uploading piece. err: <nil> Dec 29 03:48:56.207350520 (564h19m56.078433555s) [noise=true , address=145.79.227.46:4601 , piece_id=333P6ZRMBYPS3P4EIOXO5XNRSFHR7FEQZ72664KAOXWTLN7LIFGA, seg_pos={0 0}, upload=494778]: Done uploading piece. err: <nil> Dec 29 03:48:56.226214370 (564h19m56.097297405s) [noise=true , address=145.79.227.26:4655 , piece_id=4NU3DDPWAXNSZHQKGDC3UA2IS5OEI5IBHSJNLYYEEFAQH26SRF5Q, seg_pos={0 0}, upload=494778]: Done uploading piece. err: <nil> Dec 29 03:48:56.226262611 (564h19m56.097345656s) [noise=true , address=145.79.227.37:4566 , piece_id=IJ44CIMIJLNCIISU3UNOAUTSBWRCDU6Y57UZ5Q7M72IWCKX734LA, seg_pos={0 0}, upload=494778]: Done uploading piece. err: <nil> Dec 29 03:48:56.388679077 (564h19m56.259762112s) [noise=true , address=77.90.31.10:4519 , piece_id=HBIGUFSCUTHA7IENQMDZRFUEQTSFZZNNSBG4FD2NHBXBVR722PFA, seg_pos={0 0}, upload=494778]: Done uploading piece. err: <nil> Dec 29 03:48:56.394996887 (564h19m56.266079922s) [noise=true , address=77.90.31.21:4878 , piece_id=KSYB52NEH536LENGKSTDEQ2D2OAIV5W55JMPIMOZITS5Z3KNGMGQ, seg_pos={0 0}, upload=494778]: Done uploading piece. err: <nil> Dec 29 03:48:56.587038945 (564h19m56.45812198s) [noise=true , address=45.91.116.12:4225 , piece_id=QGXA7QV3CSHUF2T5SGM2VFUYCMACQJ6HYMZCAXNFV5L2UZIYY6PA, seg_pos={0 0}, upload=494778]: Done uploading piece. err: <nil> Dec 29 03:48:56.846340935 (564h19m56.717423971s) [noise=true , address=45.91.116.14:4192 , piece_id=EIF6RYZB3K6F5EGOR46V2ZOKPAFLMEYG5GIESTWHX4RT4O5L77GQ, seg_pos={0 0}, upload=494778]: Done uploading piece. err: <nil> Dec 29 03:48:57.186325409 (564h19m57.057408454s) [noise=true , address=45.91.116.10:4084 , piece_id=32I5AOGPLGKG7KTWTXY2YPEUGDYWPOZKQMZM7XTP3S24VTUKQ5ZA, seg_pos={0 0}, upload=494778]: Done uploading piece. err: <nil> Dec 29 03:48:57.186355947 (564h19m57.057438982s) [seg_pos={0 0}, upload=494778]: Segment reached optimal threshold of 13 pieces. Dec 29 03:48:57.186547543 (564h19m57.057630578s) [noise=true , address=45.91.116.5:4298 , piece_id=A47N3HMV757ODUKB4WDQL4QRIVESX6UOC4S3TPLKCPWZQRWBDUBA, seg_pos={0 0}, upload=494778]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: write tcp 188.166.225.170:54444->45.91.116.5:4298: use of closed network connection; write tcp 188.166.225.170:54444->45.91.116.5:4298: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54444->45.91.116.5:4298: use of closed network connection Dec 29 03:48:57.186635169 (564h19m57.057718214s) [noise=true , address=45.91.116.5:4298 , piece_id=A47N3HMV757ODUKB4WDQL4QRIVESX6UOC4S3TPLKCPWZQRWBDUBA, seg_pos={0 0}, upload=494778]: Long tail cancellation due to optimal threshold Dec 29 03:48:57.186826925 (564h19m57.05791s) [noise=true , address=45.91.116.6:4975 , piece_id=VPNQOMA3B7CZH66SBDADW3KLXNS7J5CV2AV4OJ5ZXQZXV4R5CFQQ, seg_pos={0 0}, upload=494778]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 29 03:48:57.186846722 (564h19m57.057929767s) [noise=true , address=45.91.116.6:4975 , piece_id=VPNQOMA3B7CZH66SBDADW3KLXNS7J5CV2AV4OJ5ZXQZXV4R5CFQQ, seg_pos={0 0}, upload=494778]: Long tail cancellation due to optimal threshold Dec 29 03:48:57.186856301 (564h19m57.057939336s) [noise=true , address=145.79.227.49:4337 , piece_id=VWXWGTHCI7NWWB237ZGJDOCMHMHZV6DV7YMIJ2LRPH4DAZ5Y6WWA, seg_pos={0 0}, upload=494778]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nTZFfzNCcVAjut7NXf9seifysJDfvXkS8aZSnXKfw6gh1TrjE): protocol: expected piece hash; context canceled; EOF Dec 29 03:48:57.186863073 (564h19m57.057946108s) [noise=true , address=145.79.227.49:4337 , piece_id=VWXWGTHCI7NWWB237ZGJDOCMHMHZV6DV7YMIJ2LRPH4DAZ5Y6WWA, seg_pos={0 0}, upload=494778]: Long tail cancellation due to optimal threshold Dec 29 03:48:57.186966791 (564h19m57.058049826s) [seg_pos={0 0}, upload=494778]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 03:48:57.187083853 (564h19m57.058166888s) [upload=494778]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 03:48:57.187090356 (564h19m57.058173391s) [upload=494778]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 03:48:57.475830730 (564h19m57.346913775s) [upload=494778]: Done uploading segments... Dec 29 03:48:57.475853523 (564h19m57.346936568s) [upload=494778]: Upload finished. err: <nil> Duration: 30.001981154s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [...........................X...................X.......X.X..] duration: 36.865046559s trace: https://jaeger.dev.storj.io/trace/262a3b4a3a5f2b2e FAIL (2025-12-29 03:48:56 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== se=true , address=64.49.13.27:4958 , piece_id=RNO7Q6EBDZOFRIDD6M4BU53OXVB52P3ZQZKC7ZQ6VEOD2AIO6MEQ, seg_pos={0 0}, upload=494748]: Uploading piece... Dec 29 03:46:39.389590023 (564h17m39.260673058s) [noise=true , address=77.90.31.11:4581 , piece_id=STAAU2BS7L5JZ2E6HW65EKZ2LTJTHB2BQT3E5IQPZ6PMFTB5R2BA, seg_pos={0 0}, upload=494748]: Uploading piece... Dec 29 03:46:39.389892929 (564h17m39.260975964s) [noise=true , address=77.90.31.16:4756 , piece_id=VH4O5TLJB2E2CI7RVAIB77AUU7T4Z7PQCFNHAHRHMWRK5U3SAU7A, seg_pos={0 0}, upload=494748]: Uploading piece... Dec 29 03:46:39.390155760 (564h17m39.261238805s) [noise=true , address=5.231.73.3:4023 , piece_id=OUGY6RHELOVD42KM3FMGBRXKRKGOUOHPY3DKSTJWGTUJ3Q5YROLQ, seg_pos={0 0}, upload=494748]: Uploading piece... Dec 29 03:46:39.390429110 (564h17m39.261512145s) [noise=true , address=145.79.227.44:4208 , piece_id=ZMYK4NBXCN3LAVJHAEUFIGIPQRPQ4IFLCPKN52XDNPQ3CKGE77NQ, seg_pos={0 0}, upload=494748]: Uploading piece... Dec 29 03:46:39.390774769 (564h17m39.261857804s) [noise=true , address=64.49.13.169:4710 , piece_id=Q27PBKCIHI36B5HIDKD6PKYVWTSI7QNV3CNEKVSSBKN3FVTPFDQA, seg_pos={0 0}, upload=494748]: Uploading piece... Dec 29 03:46:39.391067896 (564h17m39.262150931s) [noise=true , address=145.79.227.40:4068 , piece_id=6RC5TFIDPFVPJXY4QNUX4GVDCR5562GP6J5IMVMWTQKUVUVC4XAQ, seg_pos={0 0}, upload=494748]: Uploading piece... Dec 29 03:46:39.391349352 (564h17m39.262432397s) [noise=true , address=77.90.31.14:4673 , piece_id=JQMMLQPK3NY4XLMX3G4DZUV7WEFNXDWGGJ4FH3BFC5YPHYWK5L4A, seg_pos={0 0}, upload=494748]: Uploading piece... Dec 29 03:46:39.391620058 (564h17m39.262703113s) [noise=true , address=145.79.227.30:4779 , piece_id=FTSLUGXEJXOUE7MEFZWD6KJFAV6RGPWS7N7OJFSHADZZ7IIX55GQ, seg_pos={0 0}, upload=494748]: Uploading piece... Dec 29 03:46:39.391987468 (564h17m39.263070503s) [noise=true , address=77.90.31.13:4639 , piece_id=HMW2XHEVCPMULPKO7OUJYOU5BETJIXQVEXZSTY2KARXOD7HXT26Q, seg_pos={0 0}, upload=494748]: Uploading piece... Dec 29 03:46:39.392352663 (564h17m39.263435698s) [noise=true , address=77.90.31.18:4814 , piece_id=VUWCBNWEBOQP22BQVQ6M6KAHKOYJZNUIOB4ZYUZQDI7L4672NW7Q, seg_pos={0 0}, upload=494748]: Uploading piece... Dec 29 03:46:39.393422620 (564h17m39.264505656s) [noise=true , address=77.90.31.12:4604 , piece_id=FQDKYUMG2XHD4FQXGXZDTXG6MMTLNZYWDC47JJGUIOEXYTEYBMGA, seg_pos={0 0}, upload=494748]: Uploading piece... Dec 29 03:46:39.394657942 (564h17m39.265740977s) [noise=true , address=145.79.227.25:4652 , piece_id=S2HK4O5A2SDILSDE2YW6HSHC5JGD4FHMFR6GRWP6WEZJSXX6JNQA, seg_pos={0 0}, upload=494748]: Uploading piece... Dec 29 03:46:39.395699916 (564h17m39.266782951s) [noise=true , address=145.79.227.34:4925 , piece_id=XKSPDMDB2OUTOJVDJWJPCBBRLOHCL2NCKUBWK4IC43CWFC5LIJFA, seg_pos={0 0}, upload=494748]: Uploading piece... Dec 29 03:46:39.396753092 (564h17m39.267836127s) [noise=true , address=193.149.164.143:4699 , piece_id=BPV626GYTLB5KDERSZ5YMLKZ6MDJWOPK5MTXNJ4J4FU7PNHOCKIQ, seg_pos={0 0}, upload=494748]: Uploading piece... Dec 29 03:46:39.398134963 (564h17m39.269217998s) [noise=true , address=145.79.227.12:4773 , piece_id=L2O453KQ6JWGDEJT2K4VA4M2MVBSHKDJKNE7UASDJ5KPZNH4L2HA, seg_pos={0 0}, upload=494748]: Uploading piece... Dec 29 03:46:40.716827029 (564h17m40.587910064s) [upload=494748]: Next returned nil segment Dec 29 03:46:40.716839693 (564h17m40.587922728s) [upload=494748]: Waiting for error group managing segments... Dec 29 03:46:41.051033808 (564h17m40.922116843s) [noise=true , address=64.49.13.27:4958 , piece_id=RNO7Q6EBDZOFRIDD6M4BU53OXVB52P3ZQZKC7ZQ6VEOD2AIO6MEQ, seg_pos={0 0}, upload=494748]: Done uploading piece. err: <nil> Dec 29 03:46:41.262360402 (564h17m41.133443447s) [noise=true , address=145.79.227.12:4773 , piece_id=L2O453KQ6JWGDEJT2K4VA4M2MVBSHKDJKNE7UASDJ5KPZNH4L2HA, seg_pos={0 0}, upload=494748]: Done uploading piece. err: <nil> Dec 29 03:46:41.265241008 (564h17m41.136324063s) [noise=true , address=145.79.227.25:4652 , piece_id=S2HK4O5A2SDILSDE2YW6HSHC5JGD4FHMFR6GRWP6WEZJSXX6JNQA, seg_pos={0 0}, upload=494748]: Done uploading piece. err: <nil> Dec 29 03:46:41.362228264 (564h17m41.233311299s) [noise=true , address=145.79.227.46:4609 , piece_id=P3GIG3OM7H432J3IGJIBJPSDGSRQ73MPIIWE5G4UYPFZHGHRIRMA, seg_pos={0 0}, upload=494748]: Done uploading piece. err: <nil> Dec 29 03:46:41.439067256 (564h17m41.310150301s) [noise=true , address=77.90.31.11:4581 , piece_id=STAAU2BS7L5JZ2E6HW65EKZ2LTJTHB2BQT3E5IQPZ6PMFTB5R2BA, seg_pos={0 0}, upload=494748]: Done uploading piece. err: <nil> Dec 29 03:46:41.493569326 (564h17m41.364652361s) [noise=true , address=45.91.116.4:4957 , piece_id=ZQ4K2ND3IIB4AML7CIDDZETG4RZ5L2RL7D2HJZMJ3WSGJZVE3BNQ, seg_pos={0 0}, upload=494748]: Done uploading piece. err: <nil> Dec 29 03:46:41.574947141 (564h17m41.446030176s) [noise=true , address=77.90.31.12:4604 , piece_id=FQDKYUMG2XHD4FQXGXZDTXG6MMTLNZYWDC47JJGUIOEXYTEYBMGA, seg_pos={0 0}, upload=494748]: Done uploading piece. err: <nil> Dec 29 03:46:41.704651464 (564h17m41.575734499s) [noise=true , address=77.90.31.16:4756 , piece_id=VH4O5TLJB2E2CI7RVAIB77AUU7T4Z7PQCFNHAHRHMWRK5U3SAU7A, seg_pos={0 0}, upload=494748]: Done uploading piece. err: <nil> Dec 29 03:46:42.010950742 (564h17m41.882033777s) [noise=true , address=77.90.31.14:4673 , piece_id=JQMMLQPK3NY4XLMX3G4DZUV7WEFNXDWGGJ4FH3BFC5YPHYWK5L4A, seg_pos={0 0}, upload=494748]: Done uploading piece. err: <nil> Dec 29 03:46:42.172005653 (564h17m42.043088688s) [noise=true , address=145.79.227.30:4779 , piece_id=FTSLUGXEJXOUE7MEFZWD6KJFAV6RGPWS7N7OJFSHADZZ7IIX55GQ, seg_pos={0 0}, upload=494748]: Done uploading piece. err: <nil> Dec 29 03:46:42.201208108 (564h17m42.072291153s) [noise=true , address=145.79.227.44:4208 , piece_id=ZMYK4NBXCN3LAVJHAEUFIGIPQRPQ4IFLCPKN52XDNPQ3CKGE77NQ, seg_pos={0 0}, upload=494748]: Done uploading piece. err: <nil> Dec 29 03:46:42.223177351 (564h17m42.094260387s) [noise=true , address=145.79.227.34:4925 , piece_id=XKSPDMDB2OUTOJVDJWJPCBBRLOHCL2NCKUBWK4IC43CWFC5LIJFA, seg_pos={0 0}, upload=494748]: Done uploading piece. err: <nil> Dec 29 03:46:42.276938741 (564h17m42.148021786s) [noise=true , address=77.90.31.18:4814 , piece_id=VUWCBNWEBOQP22BQVQ6M6KAHKOYJZNUIOB4ZYUZQDI7L4672NW7Q, seg_pos={0 0}, upload=494748]: Done uploading piece. err: <nil> Dec 29 03:46:42.367778062 (564h17m42.238861097s) [noise=true , address=77.90.31.13:4639 , piece_id=HMW2XHEVCPMULPKO7OUJYOU5BETJIXQVEXZSTY2KARXOD7HXT26Q, seg_pos={0 0}, upload=494748]: Done uploading piece. err: <nil> Dec 29 03:46:42.367802359 (564h17m42.238885394s) [seg_pos={0 0}, upload=494748]: Segment reached optimal threshold of 14 pieces. Dec 29 03:46:42.368052876 (564h17m42.239135931s) [noise=true , address=64.49.13.169:4710 , piece_id=Q27PBKCIHI36B5HIDKD6PKYVWTSI7QNV3CNEKVSSBKN3FVTPFDQA, seg_pos={0 0}, upload=494748]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hsbGK94RFwxic9F8NKp8kKvztegUhkNmYEor4QyLJmqF3seZf): protocol: write tcp 188.166.225.170:59868->64.49.13.169:4710: use of closed network connection; write tcp 188.166.225.170:59868->64.49.13.169:4710: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59868->64.49.13.169:4710: use of closed network connection Dec 29 03:46:42.368065499 (564h17m42.239148544s) [noise=true , address=64.49.13.169:4710 , piece_id=Q27PBKCIHI36B5HIDKD6PKYVWTSI7QNV3CNEKVSSBKN3FVTPFDQA, seg_pos={0 0}, upload=494748]: Long tail cancellation due to optimal threshold Dec 29 03:46:42.368081100 (564h17m42.239164135s) [noise=true , address=5.231.73.3:4023 , piece_id=OUGY6RHELOVD42KM3FMGBRXKRKGOUOHPY3DKSTJWGTUJ3Q5YROLQ, seg_pos={0 0}, upload=494748]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17xu6xC3uRdhHYHGDMzssSg81XyXWqFT6NbC44agM2Z3mf1Jt8): protocol: expected piece hash; context canceled; EOF Dec 29 03:46:42.368087922 (564h17m42.239170957s) [noise=true , address=5.231.73.3:4023 , piece_id=OUGY6RHELOVD42KM3FMGBRXKRKGOUOHPY3DKSTJWGTUJ3Q5YROLQ, seg_pos={0 0}, upload=494748]: Long tail cancellation due to optimal threshold Dec 29 03:46:42.368237757 (564h17m42.239320792s) [noise=true , address=145.79.227.40:4068 , piece_id=6RC5TFIDPFVPJXY4QNUX4GVDCR5562GP6J5IMVMWTQKUVUVC4XAQ, seg_pos={0 0}, upload=494748]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Lh9dYpHCxbs7gpZ4Xt7os5Gm5Nco6jrNW1V1FHddVUEMkFfPY): protocol: expected piece hash; context canceled; EOF Dec 29 03:46:42.368248137 (564h17m42.239331182s) [noise=true , address=145.79.227.40:4068 , piece_id=6RC5TFIDPFVPJXY4QNUX4GVDCR5562GP6J5IMVMWTQKUVUVC4XAQ, seg_pos={0 0}, upload=494748]: Long tail cancellation due to optimal threshold Dec 29 03:46:42.368433511 (564h17m42.239516546s) [noise=true , address=193.149.164.143:4699 , piece_id=BPV626GYTLB5KDERSZ5YMLKZ6MDJWOPK5MTXNJ4J4FU7PNHOCKIQ, seg_pos={0 0}, upload=494748]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1LncNSBhhreg9JTByJPSB7eGUEu88Eyd3jc9pg24VsEjp4FULq): protocol: expected piece hash; context canceled; EOF Dec 29 03:46:42.368442537 (564h17m42.239525573s) [noise=true , address=193.149.164.143:4699 , piece_id=BPV626GYTLB5KDERSZ5YMLKZ6MDJWOPK5MTXNJ4J4FU7PNHOCKIQ, seg_pos={0 0}, upload=494748]: Long tail cancellation due to optimal threshold Dec 29 03:46:42.368532690 (564h17m42.239615725s) [seg_pos={0 0}, upload=494748]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 03:46:42.368616819 (564h17m42.239699854s) [upload=494748]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 03:46:42.368621358 (564h17m42.239704393s) [upload=494748]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 03:46:42.667817119 (564h17m42.538900164s) [upload=494748]: Done uploading segments... Dec 29 03:46:42.667836256 (564h17m42.538919301s) [upload=494748]: Upload finished. err: <nil> Duration: 36.865062609s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 7 out of 60 is failed (threshold=6) [......................XX................X......XXX......X...] duration: 5.000654s trace: https://jaeger.dev.storj.io/trace/f91d358bd5ae577 FAIL (2025-12-29 03:48:50 UTC) orbiter warning: context deadline exceeded Duration: 5.000684429s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 32 out of 60 is failed (threshold=3) [.XX.X.....X.X..XXXX..XXX.XX...X.X.....XXX...XXXXXXX.XX.XXX.X] duration: 30.004045847s trace: https://jaeger.dev.storj.io/trace/6e6e7c72daa58102 FAIL (2025-12-29 03:48:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== (564h19m15.37443602s) [upload=494767]: Done starting upload Dec 29 03:48:15.503461301 (564h19m15.374544336s) [upload=494767]: Uploading segments... Dec 29 03:48:15.503589285 (564h19m15.37467233s) [upload=494767]: Got next segment. Inline: false Dec 29 03:48:15.503620945 (564h19m15.37470398s) [upload=494767]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 03:48:15.503624722 (564h19m15.374707757s) [upload=494767]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 03:48:15.749423408 (564h19m15.620506443s) [seg_pos={0 0}, upload=494767]: Begin upload segment... Dec 29 03:48:15.749503591 (564h19m15.620586636s) [seg_pos={0 0}, upload=494767]: Done begin upload segment. Dec 29 03:48:15.749552674 (564h19m15.620635709s) [noise=true , address=145.79.227.17:4972 , piece_id=ICPJEVEBSNSQMICXXKS7BGGIJOG3OND3DVX6WAZ5A7MUD2IVMR6Q, seg_pos={0 0}, upload=494767]: Uploading piece... Dec 29 03:48:15.749573013 (564h19m15.620656048s) [noise=true , address=45.91.116.8:4045 , piece_id=TP7WY6UVARPOWTUZ337MEUUSZGEY5WUEHPODEJLQYDNGFHN6NUDA, seg_pos={0 0}, upload=494767]: Uploading piece... Dec 29 03:48:15.750353348 (564h19m15.621436393s) [noise=true , address=145.79.227.6:4259 , piece_id=OMHCIUZXKZHAAOHKMO725SB625LZKSO3TLCINQM6FRSRBAFUL45Q, seg_pos={0 0}, upload=494767]: Uploading piece... Dec 29 03:48:15.751037962 (564h19m15.622121007s) [noise=true , address=145.79.227.7:4293 , piece_id=DIBMIHYJ5GJZCJIO45TEQ5SYJP4MVXM2CJ2JKXQGECA7MGDYLBJQ, seg_pos={0 0}, upload=494767]: Uploading piece... Dec 29 03:48:15.751544838 (564h19m15.622627883s) [noise=true , address=45.91.116.7:4011 , piece_id=KPLQIU2UOEWUXN6SCNYCTVVOGVABGXTCPXNYRXF3SCPFEMRGXJJQ, seg_pos={0 0}, upload=494767]: Uploading piece... Dec 29 03:48:15.751966230 (564h19m15.623049265s) [noise=true , address=45.91.116.14:4192 , piece_id=R2NYNWNNH7JSIZOQWXHAYPBX2DPQ3DEZJCTKOZIEZOKKN5GKZ2BA, seg_pos={0 0}, upload=494767]: Uploading piece... Dec 29 03:48:15.752308212 (564h19m15.623391247s) [noise=true , address=145.79.227.45:4230 , piece_id=7GNYAVTYPOGZKO43DNHZWSISZQZOXSE4MSAMGSQRRLO2FDHF2TMQ, seg_pos={0 0}, upload=494767]: Uploading piece... Dec 29 03:48:15.753307383 (564h19m15.624390419s) [noise=true , address=45.91.116.11:4117 , piece_id=EF5ADMA6VVRECDZXPKVHLRK7DW6AZQRVFZX7EPBGE6WVTTVS6LAA, seg_pos={0 0}, upload=494767]: Uploading piece... Dec 29 03:48:15.753417303 (564h19m15.624500338s) [noise=true , address=145.79.227.32:4835 , piece_id=MRIVHJLFCETY57WJRVQUTVATQEJZ4LI2HXOLNNDM3FZTRPYJNQIQ, seg_pos={0 0}, upload=494767]: Uploading piece... Dec 29 03:48:15.754447264 (564h19m15.625530299s) [noise=true , address=145.79.227.51:4409 , piece_id=6DDNRKIICQG4IPOF465F7BW2XJMNHK7B5MAVVEOSVGANIXNRFIHA, seg_pos={0 0}, upload=494767]: Uploading piece... Dec 29 03:48:15.754494975 (564h19m15.62557801s) [noise=true , address=145.79.227.16:4909 , piece_id=4IUNG5ELZTPC6XKE7B3S6N6JKGN3S3HQQU67PXJF6P3SOHIQR2JA, seg_pos={0 0}, upload=494767]: Uploading piece... Dec 29 03:48:15.755547929 (564h19m15.626630965s) [noise=true , address=145.79.227.38:4979 , piece_id=TOKLK5FS2QQWGOVAHJ5DEJ4MHV6PS4EUXR4EL2ZC4NHURHITWVMQ, seg_pos={0 0}, upload=494767]: Uploading piece... Dec 29 03:48:15.756713399 (564h19m15.627796434s) [noise=true , address=77.90.31.10:4519 , piece_id=I6LJMBZM7E5BTLZGOKBG2VDZS4WSBIOLWU3ZDFI4XEW26QPKLOIA, seg_pos={0 0}, upload=494767]: Uploading piece... Dec 29 03:48:15.758941632 (564h19m15.630024667s) [noise=true , address=145.79.227.12:4765 , piece_id=XMFEBVL7GFFUB52TJEDITCIMC7C65YGF3WG4WQLDJPCCOZT6MRGA, seg_pos={0 0}, upload=494767]: Uploading piece... Dec 29 03:48:15.760257727 (564h19m15.631340763s) [noise=true , address=145.79.227.44:4195 , piece_id=EDX7HY5AVEABK7FOY257I3V2H5XPMVYX2ONS4HRGNOUZRBO2MN7A, seg_pos={0 0}, upload=494767]: Uploading piece... Dec 29 03:48:15.761755110 (564h19m15.632838145s) [noise=true , address=77.90.31.12:4589 , piece_id=5CEKD3EHQGGO2PDINWKVXXZ7INAZC7PVEA76HFELO5D6WLOPZMQQ, seg_pos={0 0}, upload=494767]: Uploading piece... Dec 29 03:48:17.454789200 (564h19m17.325872245s) [upload=494767]: Next returned nil segment Dec 29 03:48:17.454800502 (564h19m17.325883537s) [upload=494767]: Waiting for error group managing segments... Dec 29 03:48:17.738382142 (564h19m17.609465177s) [noise=true , address=145.79.227.51:4409 , piece_id=6DDNRKIICQG4IPOF465F7BW2XJMNHK7B5MAVVEOSVGANIXNRFIHA, seg_pos={0 0}, upload=494767]: Done uploading piece. err: <nil> Dec 29 03:48:17.739777699 (564h19m17.610860734s) [noise=true , address=145.79.227.38:4979 , piece_id=TOKLK5FS2QQWGOVAHJ5DEJ4MHV6PS4EUXR4EL2ZC4NHURHITWVMQ, seg_pos={0 0}, upload=494767]: Done uploading piece. err: <nil> Dec 29 03:48:17.741904389 (564h19m17.612987434s) [noise=true , address=145.79.227.6:4259 , piece_id=OMHCIUZXKZHAAOHKMO725SB625LZKSO3TLCINQM6FRSRBAFUL45Q, seg_pos={0 0}, upload=494767]: Done uploading piece. err: <nil> Dec 29 03:48:17.742947345 (564h19m17.61403039s) [noise=true , address=145.79.227.17:4972 , piece_id=ICPJEVEBSNSQMICXXKS7BGGIJOG3OND3DVX6WAZ5A7MUD2IVMR6Q, seg_pos={0 0}, upload=494767]: Done uploading piece. err: <nil> Dec 29 03:48:17.759716686 (564h19m17.630799721s) [noise=true , address=145.79.227.32:4835 , piece_id=MRIVHJLFCETY57WJRVQUTVATQEJZ4LI2HXOLNNDM3FZTRPYJNQIQ, seg_pos={0 0}, upload=494767]: Done uploading piece. err: <nil> Dec 29 03:48:17.761335869 (564h19m17.632418914s) [noise=true , address=145.79.227.12:4765 , piece_id=XMFEBVL7GFFUB52TJEDITCIMC7C65YGF3WG4WQLDJPCCOZT6MRGA, seg_pos={0 0}, upload=494767]: Done uploading piece. err: <nil> Dec 29 03:48:17.774801049 (564h19m17.645884084s) [noise=true , address=145.79.227.16:4909 , piece_id=4IUNG5ELZTPC6XKE7B3S6N6JKGN3S3HQQU67PXJF6P3SOHIQR2JA, seg_pos={0 0}, upload=494767]: Done uploading piece. err: <nil> Dec 29 03:48:17.777943051 (564h19m17.649026086s) [noise=true , address=145.79.227.45:4230 , piece_id=7GNYAVTYPOGZKO43DNHZWSISZQZOXSE4MSAMGSQRRLO2FDHF2TMQ, seg_pos={0 0}, upload=494767]: Done uploading piece. err: <nil> Dec 29 03:48:18.040341165 (564h19m17.9114242s) [noise=true , address=77.90.31.10:4519 , piece_id=I6LJMBZM7E5BTLZGOKBG2VDZS4WSBIOLWU3ZDFI4XEW26QPKLOIA, seg_pos={0 0}, upload=494767]: Done uploading piece. err: <nil> Dec 29 03:48:18.109493852 (564h19m17.980576897s) [noise=true , address=45.91.116.14:4192 , piece_id=R2NYNWNNH7JSIZOQWXHAYPBX2DPQ3DEZJCTKOZIEZOKKN5GKZ2BA, seg_pos={0 0}, upload=494767]: Done uploading piece. err: <nil> Dec 29 03:48:18.662085587 (564h19m18.533168622s) [noise=true , address=45.91.116.8:4045 , piece_id=TP7WY6UVARPOWTUZ337MEUUSZGEY5WUEHPODEJLQYDNGFHN6NUDA, seg_pos={0 0}, upload=494767]: Done uploading piece. err: <nil> Dec 29 03:48:18.715375698 (564h19m18.586458743s) [noise=true , address=45.91.116.7:4011 , piece_id=KPLQIU2UOEWUXN6SCNYCTVVOGVABGXTCPXNYRXF3SCPFEMRGXJJQ, seg_pos={0 0}, upload=494767]: Done uploading piece. err: <nil> Dec 29 03:48:18.779552440 (564h19m18.650635485s) [noise=true , address=45.91.116.11:4117 , piece_id=EF5ADMA6VVRECDZXPKVHLRK7DW6AZQRVFZX7EPBGE6WVTTVS6LAA, seg_pos={0 0}, upload=494767]: Done uploading piece. err: <nil> Dec 29 03:48:18.779575634 (564h19m18.650658669s) [seg_pos={0 0}, upload=494767]: Segment reached optimal threshold of 13 pieces. Dec 29 03:48:18.779805993 (564h19m18.650889028s) [noise=true , address=145.79.227.7:4293 , piece_id=DIBMIHYJ5GJZCJIO45TEQ5SYJP4MVXM2CJ2JKXQGECA7MGDYLBJQ, seg_pos={0 0}, upload=494767]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE): protocol: write tcp 188.166.225.170:38680->145.79.227.7:4293: use of closed network connection; write tcp 188.166.225.170:38680->145.79.227.7:4293: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38680->145.79.227.7:4293: use of closed network connection Dec 29 03:48:18.779815771 (564h19m18.650898816s) [noise=true , address=145.79.227.7:4293 , piece_id=DIBMIHYJ5GJZCJIO45TEQ5SYJP4MVXM2CJ2JKXQGECA7MGDYLBJQ, seg_pos={0 0}, upload=494767]: Long tail cancellation due to optimal threshold Dec 29 03:48:18.779962581 (564h19m18.651045626s) [noise=true , address=145.79.227.44:4195 , piece_id=EDX7HY5AVEABK7FOY257I3V2H5XPMVYX2ONS4HRGNOUZRBO2MN7A, seg_pos={0 0}, upload=494767]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa): protocol: expected piece hash; context canceled; EOF Dec 29 03:48:18.779972330 (564h19m18.651055365s) [noise=true , address=145.79.227.44:4195 , piece_id=EDX7HY5AVEABK7FOY257I3V2H5XPMVYX2ONS4HRGNOUZRBO2MN7A, seg_pos={0 0}, upload=494767]: Long tail cancellation due to optimal threshold Dec 29 03:48:18.780062521 (564h19m18.651145556s) [noise=true , address=77.90.31.12:4589 , piece_id=5CEKD3EHQGGO2PDINWKVXXZ7INAZC7PVEA76HFELO5D6WLOPZMQQ, seg_pos={0 0}, upload=494767]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv): protocol: read tcp 188.166.225.170:58496->77.90.31.12:4589: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 29 03:48:18.780081738 (564h19m18.651164773s) [noise=true , address=77.90.31.12:4589 , piece_id=5CEKD3EHQGGO2PDINWKVXXZ7INAZC7PVEA76HFELO5D6WLOPZMQQ, seg_pos={0 0}, upload=494767]: Long tail cancellation due to optimal threshold Dec 29 03:48:18.780113609 (564h19m18.651196644s) [seg_pos={0 0}, upload=494767]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 03:48:18.780223498 (564h19m18.651306543s) [upload=494767]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 03:48:18.780231192 (564h19m18.651314237s) [upload=494767]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 03:48:19.072162533 (564h19m18.943245568s) [upload=494767]: Done uploading segments... Dec 29 03:48:19.072181829 (564h19m18.943264864s) [upload=494767]: Upload finished. err: <nil> Duration: 30.004066497s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 7 out of 60 is failed (threshold=6) [.......................XX................X......XXX......X..] duration: 3.574452022s trace: https://jaeger.dev.storj.io/trace/f91d358bd5ae577 FAIL (2025-12-29 03:48:32 UTC) orbiter warning: context deadline exceeded Duration: 3.574468623s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [............................X...................X.......X.X.] duration: 40.427103994s trace: https://jaeger.dev.storj.io/trace/262a3b4a3a5f2b2e FAIL (2025-12-29 03:48:19 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== se=true , address=64.49.13.27:4958 , piece_id=RNO7Q6EBDZOFRIDD6M4BU53OXVB52P3ZQZKC7ZQ6VEOD2AIO6MEQ, seg_pos={0 0}, upload=494748]: Uploading piece... Dec 29 03:46:39.389590023 (564h17m39.260673058s) [noise=true , address=77.90.31.11:4581 , piece_id=STAAU2BS7L5JZ2E6HW65EKZ2LTJTHB2BQT3E5IQPZ6PMFTB5R2BA, seg_pos={0 0}, upload=494748]: Uploading piece... Dec 29 03:46:39.389892929 (564h17m39.260975964s) [noise=true , address=77.90.31.16:4756 , piece_id=VH4O5TLJB2E2CI7RVAIB77AUU7T4Z7PQCFNHAHRHMWRK5U3SAU7A, seg_pos={0 0}, upload=494748]: Uploading piece... Dec 29 03:46:39.390155760 (564h17m39.261238805s) [noise=true , address=5.231.73.3:4023 , piece_id=OUGY6RHELOVD42KM3FMGBRXKRKGOUOHPY3DKSTJWGTUJ3Q5YROLQ, seg_pos={0 0}, upload=494748]: Uploading piece... Dec 29 03:46:39.390429110 (564h17m39.261512145s) [noise=true , address=145.79.227.44:4208 , piece_id=ZMYK4NBXCN3LAVJHAEUFIGIPQRPQ4IFLCPKN52XDNPQ3CKGE77NQ, seg_pos={0 0}, upload=494748]: Uploading piece... Dec 29 03:46:39.390774769 (564h17m39.261857804s) [noise=true , address=64.49.13.169:4710 , piece_id=Q27PBKCIHI36B5HIDKD6PKYVWTSI7QNV3CNEKVSSBKN3FVTPFDQA, seg_pos={0 0}, upload=494748]: Uploading piece... Dec 29 03:46:39.391067896 (564h17m39.262150931s) [noise=true , address=145.79.227.40:4068 , piece_id=6RC5TFIDPFVPJXY4QNUX4GVDCR5562GP6J5IMVMWTQKUVUVC4XAQ, seg_pos={0 0}, upload=494748]: Uploading piece... Dec 29 03:46:39.391349352 (564h17m39.262432397s) [noise=true , address=77.90.31.14:4673 , piece_id=JQMMLQPK3NY4XLMX3G4DZUV7WEFNXDWGGJ4FH3BFC5YPHYWK5L4A, seg_pos={0 0}, upload=494748]: Uploading piece... Dec 29 03:46:39.391620058 (564h17m39.262703113s) [noise=true , address=145.79.227.30:4779 , piece_id=FTSLUGXEJXOUE7MEFZWD6KJFAV6RGPWS7N7OJFSHADZZ7IIX55GQ, seg_pos={0 0}, upload=494748]: Uploading piece... Dec 29 03:46:39.391987468 (564h17m39.263070503s) [noise=true , address=77.90.31.13:4639 , piece_id=HMW2XHEVCPMULPKO7OUJYOU5BETJIXQVEXZSTY2KARXOD7HXT26Q, seg_pos={0 0}, upload=494748]: Uploading piece... Dec 29 03:46:39.392352663 (564h17m39.263435698s) [noise=true , address=77.90.31.18:4814 , piece_id=VUWCBNWEBOQP22BQVQ6M6KAHKOYJZNUIOB4ZYUZQDI7L4672NW7Q, seg_pos={0 0}, upload=494748]: Uploading piece... Dec 29 03:46:39.393422620 (564h17m39.264505656s) [noise=true , address=77.90.31.12:4604 , piece_id=FQDKYUMG2XHD4FQXGXZDTXG6MMTLNZYWDC47JJGUIOEXYTEYBMGA, seg_pos={0 0}, upload=494748]: Uploading piece... Dec 29 03:46:39.394657942 (564h17m39.265740977s) [noise=true , address=145.79.227.25:4652 , piece_id=S2HK4O5A2SDILSDE2YW6HSHC5JGD4FHMFR6GRWP6WEZJSXX6JNQA, seg_pos={0 0}, upload=494748]: Uploading piece... Dec 29 03:46:39.395699916 (564h17m39.266782951s) [noise=true , address=145.79.227.34:4925 , piece_id=XKSPDMDB2OUTOJVDJWJPCBBRLOHCL2NCKUBWK4IC43CWFC5LIJFA, seg_pos={0 0}, upload=494748]: Uploading piece... Dec 29 03:46:39.396753092 (564h17m39.267836127s) [noise=true , address=193.149.164.143:4699 , piece_id=BPV626GYTLB5KDERSZ5YMLKZ6MDJWOPK5MTXNJ4J4FU7PNHOCKIQ, seg_pos={0 0}, upload=494748]: Uploading piece... Dec 29 03:46:39.398134963 (564h17m39.269217998s) [noise=true , address=145.79.227.12:4773 , piece_id=L2O453KQ6JWGDEJT2K4VA4M2MVBSHKDJKNE7UASDJ5KPZNH4L2HA, seg_pos={0 0}, upload=494748]: Uploading piece... Dec 29 03:46:40.716827029 (564h17m40.587910064s) [upload=494748]: Next returned nil segment Dec 29 03:46:40.716839693 (564h17m40.587922728s) [upload=494748]: Waiting for error group managing segments... Dec 29 03:46:41.051033808 (564h17m40.922116843s) [noise=true , address=64.49.13.27:4958 , piece_id=RNO7Q6EBDZOFRIDD6M4BU53OXVB52P3ZQZKC7ZQ6VEOD2AIO6MEQ, seg_pos={0 0}, upload=494748]: Done uploading piece. err: <nil> Dec 29 03:46:41.262360402 (564h17m41.133443447s) [noise=true , address=145.79.227.12:4773 , piece_id=L2O453KQ6JWGDEJT2K4VA4M2MVBSHKDJKNE7UASDJ5KPZNH4L2HA, seg_pos={0 0}, upload=494748]: Done uploading piece. err: <nil> Dec 29 03:46:41.265241008 (564h17m41.136324063s) [noise=true , address=145.79.227.25:4652 , piece_id=S2HK4O5A2SDILSDE2YW6HSHC5JGD4FHMFR6GRWP6WEZJSXX6JNQA, seg_pos={0 0}, upload=494748]: Done uploading piece. err: <nil> Dec 29 03:46:41.362228264 (564h17m41.233311299s) [noise=true , address=145.79.227.46:4609 , piece_id=P3GIG3OM7H432J3IGJIBJPSDGSRQ73MPIIWE5G4UYPFZHGHRIRMA, seg_pos={0 0}, upload=494748]: Done uploading piece. err: <nil> Dec 29 03:46:41.439067256 (564h17m41.310150301s) [noise=true , address=77.90.31.11:4581 , piece_id=STAAU2BS7L5JZ2E6HW65EKZ2LTJTHB2BQT3E5IQPZ6PMFTB5R2BA, seg_pos={0 0}, upload=494748]: Done uploading piece. err: <nil> Dec 29 03:46:41.493569326 (564h17m41.364652361s) [noise=true , address=45.91.116.4:4957 , piece_id=ZQ4K2ND3IIB4AML7CIDDZETG4RZ5L2RL7D2HJZMJ3WSGJZVE3BNQ, seg_pos={0 0}, upload=494748]: Done uploading piece. err: <nil> Dec 29 03:46:41.574947141 (564h17m41.446030176s) [noise=true , address=77.90.31.12:4604 , piece_id=FQDKYUMG2XHD4FQXGXZDTXG6MMTLNZYWDC47JJGUIOEXYTEYBMGA, seg_pos={0 0}, upload=494748]: Done uploading piece. err: <nil> Dec 29 03:46:41.704651464 (564h17m41.575734499s) [noise=true , address=77.90.31.16:4756 , piece_id=VH4O5TLJB2E2CI7RVAIB77AUU7T4Z7PQCFNHAHRHMWRK5U3SAU7A, seg_pos={0 0}, upload=494748]: Done uploading piece. err: <nil> Dec 29 03:46:42.010950742 (564h17m41.882033777s) [noise=true , address=77.90.31.14:4673 , piece_id=JQMMLQPK3NY4XLMX3G4DZUV7WEFNXDWGGJ4FH3BFC5YPHYWK5L4A, seg_pos={0 0}, upload=494748]: Done uploading piece. err: <nil> Dec 29 03:46:42.172005653 (564h17m42.043088688s) [noise=true , address=145.79.227.30:4779 , piece_id=FTSLUGXEJXOUE7MEFZWD6KJFAV6RGPWS7N7OJFSHADZZ7IIX55GQ, seg_pos={0 0}, upload=494748]: Done uploading piece. err: <nil> Dec 29 03:46:42.201208108 (564h17m42.072291153s) [noise=true , address=145.79.227.44:4208 , piece_id=ZMYK4NBXCN3LAVJHAEUFIGIPQRPQ4IFLCPKN52XDNPQ3CKGE77NQ, seg_pos={0 0}, upload=494748]: Done uploading piece. err: <nil> Dec 29 03:46:42.223177351 (564h17m42.094260387s) [noise=true , address=145.79.227.34:4925 , piece_id=XKSPDMDB2OUTOJVDJWJPCBBRLOHCL2NCKUBWK4IC43CWFC5LIJFA, seg_pos={0 0}, upload=494748]: Done uploading piece. err: <nil> Dec 29 03:46:42.276938741 (564h17m42.148021786s) [noise=true , address=77.90.31.18:4814 , piece_id=VUWCBNWEBOQP22BQVQ6M6KAHKOYJZNUIOB4ZYUZQDI7L4672NW7Q, seg_pos={0 0}, upload=494748]: Done uploading piece. err: <nil> Dec 29 03:46:42.367778062 (564h17m42.238861097s) [noise=true , address=77.90.31.13:4639 , piece_id=HMW2XHEVCPMULPKO7OUJYOU5BETJIXQVEXZSTY2KARXOD7HXT26Q, seg_pos={0 0}, upload=494748]: Done uploading piece. err: <nil> Dec 29 03:46:42.367802359 (564h17m42.238885394s) [seg_pos={0 0}, upload=494748]: Segment reached optimal threshold of 14 pieces. Dec 29 03:46:42.368052876 (564h17m42.239135931s) [noise=true , address=64.49.13.169:4710 , piece_id=Q27PBKCIHI36B5HIDKD6PKYVWTSI7QNV3CNEKVSSBKN3FVTPFDQA, seg_pos={0 0}, upload=494748]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hsbGK94RFwxic9F8NKp8kKvztegUhkNmYEor4QyLJmqF3seZf): protocol: write tcp 188.166.225.170:59868->64.49.13.169:4710: use of closed network connection; write tcp 188.166.225.170:59868->64.49.13.169:4710: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59868->64.49.13.169:4710: use of closed network connection Dec 29 03:46:42.368065499 (564h17m42.239148544s) [noise=true , address=64.49.13.169:4710 , piece_id=Q27PBKCIHI36B5HIDKD6PKYVWTSI7QNV3CNEKVSSBKN3FVTPFDQA, seg_pos={0 0}, upload=494748]: Long tail cancellation due to optimal threshold Dec 29 03:46:42.368081100 (564h17m42.239164135s) [noise=true , address=5.231.73.3:4023 , piece_id=OUGY6RHELOVD42KM3FMGBRXKRKGOUOHPY3DKSTJWGTUJ3Q5YROLQ, seg_pos={0 0}, upload=494748]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17xu6xC3uRdhHYHGDMzssSg81XyXWqFT6NbC44agM2Z3mf1Jt8): protocol: expected piece hash; context canceled; EOF Dec 29 03:46:42.368087922 (564h17m42.239170957s) [noise=true , address=5.231.73.3:4023 , piece_id=OUGY6RHELOVD42KM3FMGBRXKRKGOUOHPY3DKSTJWGTUJ3Q5YROLQ, seg_pos={0 0}, upload=494748]: Long tail cancellation due to optimal threshold Dec 29 03:46:42.368237757 (564h17m42.239320792s) [noise=true , address=145.79.227.40:4068 , piece_id=6RC5TFIDPFVPJXY4QNUX4GVDCR5562GP6J5IMVMWTQKUVUVC4XAQ, seg_pos={0 0}, upload=494748]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Lh9dYpHCxbs7gpZ4Xt7os5Gm5Nco6jrNW1V1FHddVUEMkFfPY): protocol: expected piece hash; context canceled; EOF Dec 29 03:46:42.368248137 (564h17m42.239331182s) [noise=true , address=145.79.227.40:4068 , piece_id=6RC5TFIDPFVPJXY4QNUX4GVDCR5562GP6J5IMVMWTQKUVUVC4XAQ, seg_pos={0 0}, upload=494748]: Long tail cancellation due to optimal threshold Dec 29 03:46:42.368433511 (564h17m42.239516546s) [noise=true , address=193.149.164.143:4699 , piece_id=BPV626GYTLB5KDERSZ5YMLKZ6MDJWOPK5MTXNJ4J4FU7PNHOCKIQ, seg_pos={0 0}, upload=494748]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1LncNSBhhreg9JTByJPSB7eGUEu88Eyd3jc9pg24VsEjp4FULq): protocol: expected piece hash; context canceled; EOF Dec 29 03:46:42.368442537 (564h17m42.239525573s) [noise=true , address=193.149.164.143:4699 , piece_id=BPV626GYTLB5KDERSZ5YMLKZ6MDJWOPK5MTXNJ4J4FU7PNHOCKIQ, seg_pos={0 0}, upload=494748]: Long tail cancellation due to optimal threshold Dec 29 03:46:42.368532690 (564h17m42.239615725s) [seg_pos={0 0}, upload=494748]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 03:46:42.368616819 (564h17m42.239699854s) [upload=494748]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 03:46:42.368621358 (564h17m42.239704393s) [upload=494748]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 03:46:42.667817119 (564h17m42.538900164s) [upload=494748]: Done uploading segments... Dec 29 03:46:42.667836256 (564h17m42.538919301s) [upload=494748]: Upload finished. err: <nil> Duration: 40.427121258s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [.............................X...................X.......X.X] duration: 1m0.003659589s trace: https://jaeger.dev.storj.io/trace/262a3b4a3a5f2b2e FAIL (2025-12-29 03:47:38 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== se=true , address=64.49.13.27:4958 , piece_id=RNO7Q6EBDZOFRIDD6M4BU53OXVB52P3ZQZKC7ZQ6VEOD2AIO6MEQ, seg_pos={0 0}, upload=494748]: Uploading piece... Dec 29 03:46:39.389590023 (564h17m39.260673058s) [noise=true , address=77.90.31.11:4581 , piece_id=STAAU2BS7L5JZ2E6HW65EKZ2LTJTHB2BQT3E5IQPZ6PMFTB5R2BA, seg_pos={0 0}, upload=494748]: Uploading piece... Dec 29 03:46:39.389892929 (564h17m39.260975964s) [noise=true , address=77.90.31.16:4756 , piece_id=VH4O5TLJB2E2CI7RVAIB77AUU7T4Z7PQCFNHAHRHMWRK5U3SAU7A, seg_pos={0 0}, upload=494748]: Uploading piece... Dec 29 03:46:39.390155760 (564h17m39.261238805s) [noise=true , address=5.231.73.3:4023 , piece_id=OUGY6RHELOVD42KM3FMGBRXKRKGOUOHPY3DKSTJWGTUJ3Q5YROLQ, seg_pos={0 0}, upload=494748]: Uploading piece... Dec 29 03:46:39.390429110 (564h17m39.261512145s) [noise=true , address=145.79.227.44:4208 , piece_id=ZMYK4NBXCN3LAVJHAEUFIGIPQRPQ4IFLCPKN52XDNPQ3CKGE77NQ, seg_pos={0 0}, upload=494748]: Uploading piece... Dec 29 03:46:39.390774769 (564h17m39.261857804s) [noise=true , address=64.49.13.169:4710 , piece_id=Q27PBKCIHI36B5HIDKD6PKYVWTSI7QNV3CNEKVSSBKN3FVTPFDQA, seg_pos={0 0}, upload=494748]: Uploading piece... Dec 29 03:46:39.391067896 (564h17m39.262150931s) [noise=true , address=145.79.227.40:4068 , piece_id=6RC5TFIDPFVPJXY4QNUX4GVDCR5562GP6J5IMVMWTQKUVUVC4XAQ, seg_pos={0 0}, upload=494748]: Uploading piece... Dec 29 03:46:39.391349352 (564h17m39.262432397s) [noise=true , address=77.90.31.14:4673 , piece_id=JQMMLQPK3NY4XLMX3G4DZUV7WEFNXDWGGJ4FH3BFC5YPHYWK5L4A, seg_pos={0 0}, upload=494748]: Uploading piece... Dec 29 03:46:39.391620058 (564h17m39.262703113s) [noise=true , address=145.79.227.30:4779 , piece_id=FTSLUGXEJXOUE7MEFZWD6KJFAV6RGPWS7N7OJFSHADZZ7IIX55GQ, seg_pos={0 0}, upload=494748]: Uploading piece... Dec 29 03:46:39.391987468 (564h17m39.263070503s) [noise=true , address=77.90.31.13:4639 , piece_id=HMW2XHEVCPMULPKO7OUJYOU5BETJIXQVEXZSTY2KARXOD7HXT26Q, seg_pos={0 0}, upload=494748]: Uploading piece... Dec 29 03:46:39.392352663 (564h17m39.263435698s) [noise=true , address=77.90.31.18:4814 , piece_id=VUWCBNWEBOQP22BQVQ6M6KAHKOYJZNUIOB4ZYUZQDI7L4672NW7Q, seg_pos={0 0}, upload=494748]: Uploading piece... Dec 29 03:46:39.393422620 (564h17m39.264505656s) [noise=true , address=77.90.31.12:4604 , piece_id=FQDKYUMG2XHD4FQXGXZDTXG6MMTLNZYWDC47JJGUIOEXYTEYBMGA, seg_pos={0 0}, upload=494748]: Uploading piece... Dec 29 03:46:39.394657942 (564h17m39.265740977s) [noise=true , address=145.79.227.25:4652 , piece_id=S2HK4O5A2SDILSDE2YW6HSHC5JGD4FHMFR6GRWP6WEZJSXX6JNQA, seg_pos={0 0}, upload=494748]: Uploading piece... Dec 29 03:46:39.395699916 (564h17m39.266782951s) [noise=true , address=145.79.227.34:4925 , piece_id=XKSPDMDB2OUTOJVDJWJPCBBRLOHCL2NCKUBWK4IC43CWFC5LIJFA, seg_pos={0 0}, upload=494748]: Uploading piece... Dec 29 03:46:39.396753092 (564h17m39.267836127s) [noise=true , address=193.149.164.143:4699 , piece_id=BPV626GYTLB5KDERSZ5YMLKZ6MDJWOPK5MTXNJ4J4FU7PNHOCKIQ, seg_pos={0 0}, upload=494748]: Uploading piece... Dec 29 03:46:39.398134963 (564h17m39.269217998s) [noise=true , address=145.79.227.12:4773 , piece_id=L2O453KQ6JWGDEJT2K4VA4M2MVBSHKDJKNE7UASDJ5KPZNH4L2HA, seg_pos={0 0}, upload=494748]: Uploading piece... Dec 29 03:46:40.716827029 (564h17m40.587910064s) [upload=494748]: Next returned nil segment Dec 29 03:46:40.716839693 (564h17m40.587922728s) [upload=494748]: Waiting for error group managing segments... Dec 29 03:46:41.051033808 (564h17m40.922116843s) [noise=true , address=64.49.13.27:4958 , piece_id=RNO7Q6EBDZOFRIDD6M4BU53OXVB52P3ZQZKC7ZQ6VEOD2AIO6MEQ, seg_pos={0 0}, upload=494748]: Done uploading piece. err: <nil> Dec 29 03:46:41.262360402 (564h17m41.133443447s) [noise=true , address=145.79.227.12:4773 , piece_id=L2O453KQ6JWGDEJT2K4VA4M2MVBSHKDJKNE7UASDJ5KPZNH4L2HA, seg_pos={0 0}, upload=494748]: Done uploading piece. err: <nil> Dec 29 03:46:41.265241008 (564h17m41.136324063s) [noise=true , address=145.79.227.25:4652 , piece_id=S2HK4O5A2SDILSDE2YW6HSHC5JGD4FHMFR6GRWP6WEZJSXX6JNQA, seg_pos={0 0}, upload=494748]: Done uploading piece. err: <nil> Dec 29 03:46:41.362228264 (564h17m41.233311299s) [noise=true , address=145.79.227.46:4609 , piece_id=P3GIG3OM7H432J3IGJIBJPSDGSRQ73MPIIWE5G4UYPFZHGHRIRMA, seg_pos={0 0}, upload=494748]: Done uploading piece. err: <nil> Dec 29 03:46:41.439067256 (564h17m41.310150301s) [noise=true , address=77.90.31.11:4581 , piece_id=STAAU2BS7L5JZ2E6HW65EKZ2LTJTHB2BQT3E5IQPZ6PMFTB5R2BA, seg_pos={0 0}, upload=494748]: Done uploading piece. err: <nil> Dec 29 03:46:41.493569326 (564h17m41.364652361s) [noise=true , address=45.91.116.4:4957 , piece_id=ZQ4K2ND3IIB4AML7CIDDZETG4RZ5L2RL7D2HJZMJ3WSGJZVE3BNQ, seg_pos={0 0}, upload=494748]: Done uploading piece. err: <nil> Dec 29 03:46:41.574947141 (564h17m41.446030176s) [noise=true , address=77.90.31.12:4604 , piece_id=FQDKYUMG2XHD4FQXGXZDTXG6MMTLNZYWDC47JJGUIOEXYTEYBMGA, seg_pos={0 0}, upload=494748]: Done uploading piece. err: <nil> Dec 29 03:46:41.704651464 (564h17m41.575734499s) [noise=true , address=77.90.31.16:4756 , piece_id=VH4O5TLJB2E2CI7RVAIB77AUU7T4Z7PQCFNHAHRHMWRK5U3SAU7A, seg_pos={0 0}, upload=494748]: Done uploading piece. err: <nil> Dec 29 03:46:42.010950742 (564h17m41.882033777s) [noise=true , address=77.90.31.14:4673 , piece_id=JQMMLQPK3NY4XLMX3G4DZUV7WEFNXDWGGJ4FH3BFC5YPHYWK5L4A, seg_pos={0 0}, upload=494748]: Done uploading piece. err: <nil> Dec 29 03:46:42.172005653 (564h17m42.043088688s) [noise=true , address=145.79.227.30:4779 , piece_id=FTSLUGXEJXOUE7MEFZWD6KJFAV6RGPWS7N7OJFSHADZZ7IIX55GQ, seg_pos={0 0}, upload=494748]: Done uploading piece. err: <nil> Dec 29 03:46:42.201208108 (564h17m42.072291153s) [noise=true , address=145.79.227.44:4208 , piece_id=ZMYK4NBXCN3LAVJHAEUFIGIPQRPQ4IFLCPKN52XDNPQ3CKGE77NQ, seg_pos={0 0}, upload=494748]: Done uploading piece. err: <nil> Dec 29 03:46:42.223177351 (564h17m42.094260387s) [noise=true , address=145.79.227.34:4925 , piece_id=XKSPDMDB2OUTOJVDJWJPCBBRLOHCL2NCKUBWK4IC43CWFC5LIJFA, seg_pos={0 0}, upload=494748]: Done uploading piece. err: <nil> Dec 29 03:46:42.276938741 (564h17m42.148021786s) [noise=true , address=77.90.31.18:4814 , piece_id=VUWCBNWEBOQP22BQVQ6M6KAHKOYJZNUIOB4ZYUZQDI7L4672NW7Q, seg_pos={0 0}, upload=494748]: Done uploading piece. err: <nil> Dec 29 03:46:42.367778062 (564h17m42.238861097s) [noise=true , address=77.90.31.13:4639 , piece_id=HMW2XHEVCPMULPKO7OUJYOU5BETJIXQVEXZSTY2KARXOD7HXT26Q, seg_pos={0 0}, upload=494748]: Done uploading piece. err: <nil> Dec 29 03:46:42.367802359 (564h17m42.238885394s) [seg_pos={0 0}, upload=494748]: Segment reached optimal threshold of 14 pieces. Dec 29 03:46:42.368052876 (564h17m42.239135931s) [noise=true , address=64.49.13.169:4710 , piece_id=Q27PBKCIHI36B5HIDKD6PKYVWTSI7QNV3CNEKVSSBKN3FVTPFDQA, seg_pos={0 0}, upload=494748]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hsbGK94RFwxic9F8NKp8kKvztegUhkNmYEor4QyLJmqF3seZf): protocol: write tcp 188.166.225.170:59868->64.49.13.169:4710: use of closed network connection; write tcp 188.166.225.170:59868->64.49.13.169:4710: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59868->64.49.13.169:4710: use of closed network connection Dec 29 03:46:42.368065499 (564h17m42.239148544s) [noise=true , address=64.49.13.169:4710 , piece_id=Q27PBKCIHI36B5HIDKD6PKYVWTSI7QNV3CNEKVSSBKN3FVTPFDQA, seg_pos={0 0}, upload=494748]: Long tail cancellation due to optimal threshold Dec 29 03:46:42.368081100 (564h17m42.239164135s) [noise=true , address=5.231.73.3:4023 , piece_id=OUGY6RHELOVD42KM3FMGBRXKRKGOUOHPY3DKSTJWGTUJ3Q5YROLQ, seg_pos={0 0}, upload=494748]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17xu6xC3uRdhHYHGDMzssSg81XyXWqFT6NbC44agM2Z3mf1Jt8): protocol: expected piece hash; context canceled; EOF Dec 29 03:46:42.368087922 (564h17m42.239170957s) [noise=true , address=5.231.73.3:4023 , piece_id=OUGY6RHELOVD42KM3FMGBRXKRKGOUOHPY3DKSTJWGTUJ3Q5YROLQ, seg_pos={0 0}, upload=494748]: Long tail cancellation due to optimal threshold Dec 29 03:46:42.368237757 (564h17m42.239320792s) [noise=true , address=145.79.227.40:4068 , piece_id=6RC5TFIDPFVPJXY4QNUX4GVDCR5562GP6J5IMVMWTQKUVUVC4XAQ, seg_pos={0 0}, upload=494748]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Lh9dYpHCxbs7gpZ4Xt7os5Gm5Nco6jrNW1V1FHddVUEMkFfPY): protocol: expected piece hash; context canceled; EOF Dec 29 03:46:42.368248137 (564h17m42.239331182s) [noise=true , address=145.79.227.40:4068 , piece_id=6RC5TFIDPFVPJXY4QNUX4GVDCR5562GP6J5IMVMWTQKUVUVC4XAQ, seg_pos={0 0}, upload=494748]: Long tail cancellation due to optimal threshold Dec 29 03:46:42.368433511 (564h17m42.239516546s) [noise=true , address=193.149.164.143:4699 , piece_id=BPV626GYTLB5KDERSZ5YMLKZ6MDJWOPK5MTXNJ4J4FU7PNHOCKIQ, seg_pos={0 0}, upload=494748]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1LncNSBhhreg9JTByJPSB7eGUEu88Eyd3jc9pg24VsEjp4FULq): protocol: expected piece hash; context canceled; EOF Dec 29 03:46:42.368442537 (564h17m42.239525573s) [noise=true , address=193.149.164.143:4699 , piece_id=BPV626GYTLB5KDERSZ5YMLKZ6MDJWOPK5MTXNJ4J4FU7PNHOCKIQ, seg_pos={0 0}, upload=494748]: Long tail cancellation due to optimal threshold Dec 29 03:46:42.368532690 (564h17m42.239615725s) [seg_pos={0 0}, upload=494748]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 03:46:42.368616819 (564h17m42.239699854s) [upload=494748]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 03:46:42.368621358 (564h17m42.239704393s) [upload=494748]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 03:46:42.667817119 (564h17m42.538900164s) [upload=494748]: Done uploading segments... Dec 29 03:46:42.667836256 (564h17m42.538919301s) [upload=494748]: Upload finished. err: <nil> Duration: 1m0.003673155s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 7 out of 60 is failed (threshold=6) [........................XX................X......XXX......X.] duration: 3.636422156s trace: https://jaeger.dev.storj.io/trace/f91d358bd5ae577 FAIL (2025-12-29 03:47:23 UTC) orbiter warning: context deadline exceeded Duration: 3.63644559s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 31 out of 60 is failed (threshold=3) [..XX.X.....X.X..XXXX..XXX.XX...X.X.....XXX...XXXXXXX.XX.XXX.] duration: 5.650263732s trace: https://jaeger.dev.storj.io/trace/1aac2aabddff8776 FAIL (2025-12-29 03:47:10 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, Piece ID: UTIIPUBHYRBXHKPJSAFKS5ZO6SMHA776ZJNL4BWPBAAUNMZTBSAA) context deadline exceeded; piecestore: (Node ID: 1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, Piece ID: 6PZJDOZHBMD75ISQNWXBFXCIZAN7MEUM62BCRXX6PHJJBXS7BASA) context deadline exceeded; piecestore: (Node ID: 122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd, Piece ID: ESPZPLPCADOSB7RTOG7FHKOJGSHLXOJ5PLUKJRPJ7XXZR6JZW5MQ) context deadline exceeded ======LOGS====== c 29 03:46:19.927844198 (564h17m19.798927233s) [upload=494742]: Starting upload Dec 29 03:46:19.927912628 (564h17m19.798995663s) [upload=494742]: Done starting upload Dec 29 03:46:19.928136675 (564h17m19.79921972s) [upload=494742]: Uploading segments... Dec 29 03:46:19.928250852 (564h17m19.799333888s) [upload=494742]: Got next segment. Inline: false Dec 29 03:46:19.928296399 (564h17m19.799379434s) [upload=494742]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 03:46:19.928303273 (564h17m19.799386308s) [upload=494742]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 03:46:20.174879539 (564h17m20.045962584s) [seg_pos={0 0}, upload=494742]: Begin upload segment... Dec 29 03:46:20.174936057 (564h17m20.046019092s) [seg_pos={0 0}, upload=494742]: Done begin upload segment. Dec 29 03:46:20.174967597 (564h17m20.046050632s) [noise=true , address=145.79.227.12:4767 , piece_id=N4JL43ZKUXYA5M2OIXJTIWDJRO4VQII343ZMWLAHAPVBMXEXZWMA, seg_pos={0 0}, upload=494742]: Uploading piece... Dec 29 03:46:20.175133262 (564h17m20.046216297s) [noise=true , address=145.79.227.32:4835 , piece_id=RLI3XHCNEAJT6NC3AGBPDB2CGNPGOQL4IJCTGP4MFSW37BRQKPPA, seg_pos={0 0}, upload=494742]: Uploading piece... Dec 29 03:46:20.176039227 (564h17m20.047122262s) [noise=true , address=45.91.116.7:4010 , piece_id=EQGTWFZXAUEWVCABN6F7MM2GVANVSLREDBR2I5NBTA6YGHZPTUSQ, seg_pos={0 0}, upload=494742]: Uploading piece... Dec 29 03:46:20.176212487 (564h17m20.047295522s) [noise=true , address=45.91.116.14:4192 , piece_id=UTIIPUBHYRBXHKPJSAFKS5ZO6SMHA776ZJNL4BWPBAAUNMZTBSAA, seg_pos={0 0}, upload=494742]: Uploading piece... Dec 29 03:46:20.176497420 (564h17m20.047580466s) [noise=true , address=45.91.116.13:4156 , piece_id=6PZJDOZHBMD75ISQNWXBFXCIZAN7MEUM62BCRXX6PHJJBXS7BASA, seg_pos={0 0}, upload=494742]: Uploading piece... Dec 29 03:46:20.176606478 (564h17m20.047689513s) [noise=true , address=145.79.227.47:4267 , piece_id=G3ATHA3JGMLQW7GDFAZJNULN35Z2JKHHKSW76Y7ERSY3UHGS3OZQ, seg_pos={0 0}, upload=494742]: Uploading piece... Dec 29 03:46:20.176963748 (564h17m20.048046793s) [noise=true , address=45.91.116.4:4937 , piece_id=OCSN7SG4ET6EEVKNYJZZ3COK36Z6FGGK5UBFR64ZLUJHOZUTQQKA, seg_pos={0 0}, upload=494742]: Uploading piece... Dec 29 03:46:20.176995398 (564h17m20.048078433s) [noise=true , address=145.79.227.11:4732 , piece_id=YF7IT7O2RP6BYQCUXDPAEDTES4Y6QGPL2MEWPZUO2EK7Q6BLZAZQ, seg_pos={0 0}, upload=494742]: Uploading piece... Dec 29 03:46:20.177380632 (564h17m20.048463667s) [noise=true , address=77.90.31.17:4773 , piece_id=GVJT4ZD3WZH5PVUBRS7CIJAIOXL27Y4ZOSHYB2MUIGDUQYJERGRQ, seg_pos={0 0}, upload=494742]: Uploading piece... Dec 29 03:46:20.177466585 (564h17m20.04854962s) [noise=true , address=145.79.227.51:4411 , piece_id=ZQMLVHZWLZNQ2J73XBZN6OQXB3RTJMDEEFGY5AZUEURFBOJDXOUQ, seg_pos={0 0}, upload=494742]: Uploading piece... Dec 29 03:46:20.177969764 (564h17m20.049052799s) [noise=true , address=145.79.227.17:4971 , piece_id=3UFDHMLOTFVJFWAW4YMZD3WEWAFP2WVG753HOW5LZCCBJMP3RQQQ, seg_pos={0 0}, upload=494742]: Uploading piece... Dec 29 03:46:20.178729430 (564h17m20.049812466s) [noise=true , address=45.91.116.11:4117 , piece_id=DOG2IOG5JXUMB4EQB4L2VUGHOUPE7623MSPST52XVTBKXBQUERXA, seg_pos={0 0}, upload=494742]: Uploading piece... Dec 29 03:46:20.179221818 (564h17m20.050304853s) [noise=true , address=145.79.227.39:4014 , piece_id=ESPZPLPCADOSB7RTOG7FHKOJGSHLXOJ5PLUKJRPJ7XXZR6JZW5MQ, seg_pos={0 0}, upload=494742]: Uploading piece... Dec 29 03:46:20.180048623 (564h17m20.051131658s) [noise=true , address=145.79.227.48:4301 , piece_id=U7JKK2Z3NA7LKHPMY67NRVTN3LLIONMLQQ7MKPLJKISWWBOF32UQ, seg_pos={0 0}, upload=494742]: Uploading piece... Dec 29 03:46:20.180619910 (564h17m20.051702955s) [noise=true , address=145.79.227.35:4483 , piece_id=W7VC2I6KFENTE24ADU3MG6YZ7WBSM26T255IG4H7SMYULPBNUZHQ, seg_pos={0 0}, upload=494742]: Uploading piece... Dec 29 03:46:20.181068895 (564h17m20.05215193s) [noise=true , address=77.90.31.11:4554 , piece_id=2JKGCPJ7XOCPBAYHL6EKTJ5XQGSZQ66JSJBOBICMFU6TQ46RUX4Q, seg_pos={0 0}, upload=494742]: Uploading piece... Dec 29 03:46:21.870145118 (564h17m21.741228153s) [upload=494742]: Next returned nil segment Dec 29 03:46:21.870160507 (564h17m21.741243542s) [upload=494742]: Waiting for error group managing segments... Dec 29 03:46:22.157366298 (564h17m22.028449343s) [noise=true , address=145.79.227.51:4411 , piece_id=ZQMLVHZWLZNQ2J73XBZN6OQXB3RTJMDEEFGY5AZUEURFBOJDXOUQ, seg_pos={0 0}, upload=494742]: Done uploading piece. err: <nil> Dec 29 03:46:22.158994128 (564h17m22.030077163s) [noise=true , address=145.79.227.12:4767 , piece_id=N4JL43ZKUXYA5M2OIXJTIWDJRO4VQII343ZMWLAHAPVBMXEXZWMA, seg_pos={0 0}, upload=494742]: Done uploading piece. err: <nil> Dec 29 03:46:22.165853730 (564h17m22.036936765s) [noise=true , address=145.79.227.17:4971 , piece_id=3UFDHMLOTFVJFWAW4YMZD3WEWAFP2WVG753HOW5LZCCBJMP3RQQQ, seg_pos={0 0}, upload=494742]: Done uploading piece. err: <nil> Dec 29 03:46:22.167710024 (564h17m22.038793069s) [noise=true , address=145.79.227.48:4301 , piece_id=U7JKK2Z3NA7LKHPMY67NRVTN3LLIONMLQQ7MKPLJKISWWBOF32UQ, seg_pos={0 0}, upload=494742]: Done uploading piece. err: <nil> Dec 29 03:46:22.171214316 (564h17m22.042297351s) [noise=true , address=145.79.227.47:4267 , piece_id=G3ATHA3JGMLQW7GDFAZJNULN35Z2JKHHKSW76Y7ERSY3UHGS3OZQ, seg_pos={0 0}, upload=494742]: Done uploading piece. err: <nil> Dec 29 03:46:22.172234920 (564h17m22.043317955s) [noise=true , address=145.79.227.11:4732 , piece_id=YF7IT7O2RP6BYQCUXDPAEDTES4Y6QGPL2MEWPZUO2EK7Q6BLZAZQ, seg_pos={0 0}, upload=494742]: Done uploading piece. err: <nil> Dec 29 03:46:22.173933193 (564h17m22.045016228s) [noise=true , address=145.79.227.39:4014 , piece_id=ESPZPLPCADOSB7RTOG7FHKOJGSHLXOJ5PLUKJRPJ7XXZR6JZW5MQ, seg_pos={0 0}, upload=494742]: Done uploading piece. err: <nil> Dec 29 03:46:22.347160674 (564h17m22.218243709s) [noise=true , address=145.79.227.32:4835 , piece_id=RLI3XHCNEAJT6NC3AGBPDB2CGNPGOQL4IJCTGP4MFSW37BRQKPPA, seg_pos={0 0}, upload=494742]: Done uploading piece. err: <nil> Dec 29 03:46:22.523668119 (564h17m22.394751164s) [noise=true , address=45.91.116.4:4937 , piece_id=OCSN7SG4ET6EEVKNYJZZ3COK36Z6FGGK5UBFR64ZLUJHOZUTQQKA, seg_pos={0 0}, upload=494742]: Done uploading piece. err: <nil> Dec 29 03:46:22.548878383 (564h17m22.419961418s) [noise=true , address=45.91.116.14:4192 , piece_id=UTIIPUBHYRBXHKPJSAFKS5ZO6SMHA776ZJNL4BWPBAAUNMZTBSAA, seg_pos={0 0}, upload=494742]: Done uploading piece. err: <nil> Dec 29 03:46:22.577987642 (564h17m22.449070677s) [noise=true , address=45.91.116.11:4117 , piece_id=DOG2IOG5JXUMB4EQB4L2VUGHOUPE7623MSPST52XVTBKXBQUERXA, seg_pos={0 0}, upload=494742]: Done uploading piece. err: <nil> Dec 29 03:46:22.778288333 (564h17m22.649371368s) [noise=true , address=45.91.116.13:4156 , piece_id=6PZJDOZHBMD75ISQNWXBFXCIZAN7MEUM62BCRXX6PHJJBXS7BASA, seg_pos={0 0}, upload=494742]: Done uploading piece. err: <nil> Dec 29 03:46:23.343786201 (564h17m23.214869246s) [noise=true , address=45.91.116.7:4010 , piece_id=EQGTWFZXAUEWVCABN6F7MM2GVANVSLREDBR2I5NBTA6YGHZPTUSQ, seg_pos={0 0}, upload=494742]: Done uploading piece. err: <nil> Dec 29 03:46:23.343810318 (564h17m23.214893353s) [seg_pos={0 0}, upload=494742]: Segment reached optimal threshold of 13 pieces. Dec 29 03:46:23.344039785 (564h17m23.21512283s) [noise=true , address=77.90.31.11:4554 , piece_id=2JKGCPJ7XOCPBAYHL6EKTJ5XQGSZQ66JSJBOBICMFU6TQ46RUX4Q, seg_pos={0 0}, upload=494742]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: expected piece hash; context canceled; EOF Dec 29 03:46:23.344049593 (564h17m23.215132658s) [noise=true , address=77.90.31.11:4554 , piece_id=2JKGCPJ7XOCPBAYHL6EKTJ5XQGSZQ66JSJBOBICMFU6TQ46RUX4Q, seg_pos={0 0}, upload=494742]: Long tail cancellation due to optimal threshold Dec 29 03:46:23.344275513 (564h17m23.215358558s) [noise=true , address=77.90.31.17:4773 , piece_id=GVJT4ZD3WZH5PVUBRS7CIJAIOXL27Y4ZOSHYB2MUIGDUQYJERGRQ, seg_pos={0 0}, upload=494742]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ): protocol: write tcp 188.166.225.170:40866->77.90.31.17:4773: use of closed network connection; write tcp 188.166.225.170:40866->77.90.31.17:4773: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40866->77.90.31.17:4773: use of closed network connection Dec 29 03:46:23.344284691 (564h17m23.215367726s) [noise=true , address=77.90.31.17:4773 , piece_id=GVJT4ZD3WZH5PVUBRS7CIJAIOXL27Y4ZOSHYB2MUIGDUQYJERGRQ, seg_pos={0 0}, upload=494742]: Long tail cancellation due to optimal threshold Dec 29 03:46:23.344635579 (564h17m23.215718634s) [noise=true , address=145.79.227.35:4483 , piece_id=W7VC2I6KFENTE24ADU3MG6YZ7WBSM26T255IG4H7SMYULPBNUZHQ, seg_pos={0 0}, upload=494742]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1igSs2ZGX3zWVsSdmm1pnwYd7JckcyBubTuNU1gJDxQsZ9nKaY): protocol: expected piece hash; context canceled; EOF Dec 29 03:46:23.344656268 (564h17m23.215739313s) [noise=true , address=145.79.227.35:4483 , piece_id=W7VC2I6KFENTE24ADU3MG6YZ7WBSM26T255IG4H7SMYULPBNUZHQ, seg_pos={0 0}, upload=494742]: Long tail cancellation due to optimal threshold Dec 29 03:46:23.344695913 (564h17m23.215778948s) [seg_pos={0 0}, upload=494742]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 03:46:23.344816203 (564h17m23.215899238s) [upload=494742]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 03:46:23.344824178 (564h17m23.215907223s) [upload=494742]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 03:46:23.642920316 (564h17m23.514003351s) [upload=494742]: Done uploading segments... Dec 29 03:46:23.642942768 (564h17m23.514025803s) [upload=494742]: Upload finished. err: <nil> Duration: 5.650289632s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 31 out of 60 is failed (threshold=3) [...XX.X.....X.X..XXXX..XXX.XX...X.X.....XXX...XXXXXXX.XX.XXX] duration: 30.00301753s trace: https://jaeger.dev.storj.io/trace/1aac2aabddff8776 FAIL (2025-12-29 03:46:49 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, Piece ID: UTIIPUBHYRBXHKPJSAFKS5ZO6SMHA776ZJNL4BWPBAAUNMZTBSAA) context deadline exceeded; piecestore: (Node ID: 1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, Piece ID: 6PZJDOZHBMD75ISQNWXBFXCIZAN7MEUM62BCRXX6PHJJBXS7BASA) context deadline exceeded; piecestore: (Node ID: 122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd, Piece ID: ESPZPLPCADOSB7RTOG7FHKOJGSHLXOJ5PLUKJRPJ7XXZR6JZW5MQ) context deadline exceeded ======LOGS====== c 29 03:46:19.927844198 (564h17m19.798927233s) [upload=494742]: Starting upload Dec 29 03:46:19.927912628 (564h17m19.798995663s) [upload=494742]: Done starting upload Dec 29 03:46:19.928136675 (564h17m19.79921972s) [upload=494742]: Uploading segments... Dec 29 03:46:19.928250852 (564h17m19.799333888s) [upload=494742]: Got next segment. Inline: false Dec 29 03:46:19.928296399 (564h17m19.799379434s) [upload=494742]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 03:46:19.928303273 (564h17m19.799386308s) [upload=494742]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 03:46:20.174879539 (564h17m20.045962584s) [seg_pos={0 0}, upload=494742]: Begin upload segment... Dec 29 03:46:20.174936057 (564h17m20.046019092s) [seg_pos={0 0}, upload=494742]: Done begin upload segment. Dec 29 03:46:20.174967597 (564h17m20.046050632s) [noise=true , address=145.79.227.12:4767 , piece_id=N4JL43ZKUXYA5M2OIXJTIWDJRO4VQII343ZMWLAHAPVBMXEXZWMA, seg_pos={0 0}, upload=494742]: Uploading piece... Dec 29 03:46:20.175133262 (564h17m20.046216297s) [noise=true , address=145.79.227.32:4835 , piece_id=RLI3XHCNEAJT6NC3AGBPDB2CGNPGOQL4IJCTGP4MFSW37BRQKPPA, seg_pos={0 0}, upload=494742]: Uploading piece... Dec 29 03:46:20.176039227 (564h17m20.047122262s) [noise=true , address=45.91.116.7:4010 , piece_id=EQGTWFZXAUEWVCABN6F7MM2GVANVSLREDBR2I5NBTA6YGHZPTUSQ, seg_pos={0 0}, upload=494742]: Uploading piece... Dec 29 03:46:20.176212487 (564h17m20.047295522s) [noise=true , address=45.91.116.14:4192 , piece_id=UTIIPUBHYRBXHKPJSAFKS5ZO6SMHA776ZJNL4BWPBAAUNMZTBSAA, seg_pos={0 0}, upload=494742]: Uploading piece... Dec 29 03:46:20.176497420 (564h17m20.047580466s) [noise=true , address=45.91.116.13:4156 , piece_id=6PZJDOZHBMD75ISQNWXBFXCIZAN7MEUM62BCRXX6PHJJBXS7BASA, seg_pos={0 0}, upload=494742]: Uploading piece... Dec 29 03:46:20.176606478 (564h17m20.047689513s) [noise=true , address=145.79.227.47:4267 , piece_id=G3ATHA3JGMLQW7GDFAZJNULN35Z2JKHHKSW76Y7ERSY3UHGS3OZQ, seg_pos={0 0}, upload=494742]: Uploading piece... Dec 29 03:46:20.176963748 (564h17m20.048046793s) [noise=true , address=45.91.116.4:4937 , piece_id=OCSN7SG4ET6EEVKNYJZZ3COK36Z6FGGK5UBFR64ZLUJHOZUTQQKA, seg_pos={0 0}, upload=494742]: Uploading piece... Dec 29 03:46:20.176995398 (564h17m20.048078433s) [noise=true , address=145.79.227.11:4732 , piece_id=YF7IT7O2RP6BYQCUXDPAEDTES4Y6QGPL2MEWPZUO2EK7Q6BLZAZQ, seg_pos={0 0}, upload=494742]: Uploading piece... Dec 29 03:46:20.177380632 (564h17m20.048463667s) [noise=true , address=77.90.31.17:4773 , piece_id=GVJT4ZD3WZH5PVUBRS7CIJAIOXL27Y4ZOSHYB2MUIGDUQYJERGRQ, seg_pos={0 0}, upload=494742]: Uploading piece... Dec 29 03:46:20.177466585 (564h17m20.04854962s) [noise=true , address=145.79.227.51:4411 , piece_id=ZQMLVHZWLZNQ2J73XBZN6OQXB3RTJMDEEFGY5AZUEURFBOJDXOUQ, seg_pos={0 0}, upload=494742]: Uploading piece... Dec 29 03:46:20.177969764 (564h17m20.049052799s) [noise=true , address=145.79.227.17:4971 , piece_id=3UFDHMLOTFVJFWAW4YMZD3WEWAFP2WVG753HOW5LZCCBJMP3RQQQ, seg_pos={0 0}, upload=494742]: Uploading piece... Dec 29 03:46:20.178729430 (564h17m20.049812466s) [noise=true , address=45.91.116.11:4117 , piece_id=DOG2IOG5JXUMB4EQB4L2VUGHOUPE7623MSPST52XVTBKXBQUERXA, seg_pos={0 0}, upload=494742]: Uploading piece... Dec 29 03:46:20.179221818 (564h17m20.050304853s) [noise=true , address=145.79.227.39:4014 , piece_id=ESPZPLPCADOSB7RTOG7FHKOJGSHLXOJ5PLUKJRPJ7XXZR6JZW5MQ, seg_pos={0 0}, upload=494742]: Uploading piece... Dec 29 03:46:20.180048623 (564h17m20.051131658s) [noise=true , address=145.79.227.48:4301 , piece_id=U7JKK2Z3NA7LKHPMY67NRVTN3LLIONMLQQ7MKPLJKISWWBOF32UQ, seg_pos={0 0}, upload=494742]: Uploading piece... Dec 29 03:46:20.180619910 (564h17m20.051702955s) [noise=true , address=145.79.227.35:4483 , piece_id=W7VC2I6KFENTE24ADU3MG6YZ7WBSM26T255IG4H7SMYULPBNUZHQ, seg_pos={0 0}, upload=494742]: Uploading piece... Dec 29 03:46:20.181068895 (564h17m20.05215193s) [noise=true , address=77.90.31.11:4554 , piece_id=2JKGCPJ7XOCPBAYHL6EKTJ5XQGSZQ66JSJBOBICMFU6TQ46RUX4Q, seg_pos={0 0}, upload=494742]: Uploading piece... Dec 29 03:46:21.870145118 (564h17m21.741228153s) [upload=494742]: Next returned nil segment Dec 29 03:46:21.870160507 (564h17m21.741243542s) [upload=494742]: Waiting for error group managing segments... Dec 29 03:46:22.157366298 (564h17m22.028449343s) [noise=true , address=145.79.227.51:4411 , piece_id=ZQMLVHZWLZNQ2J73XBZN6OQXB3RTJMDEEFGY5AZUEURFBOJDXOUQ, seg_pos={0 0}, upload=494742]: Done uploading piece. err: <nil> Dec 29 03:46:22.158994128 (564h17m22.030077163s) [noise=true , address=145.79.227.12:4767 , piece_id=N4JL43ZKUXYA5M2OIXJTIWDJRO4VQII343ZMWLAHAPVBMXEXZWMA, seg_pos={0 0}, upload=494742]: Done uploading piece. err: <nil> Dec 29 03:46:22.165853730 (564h17m22.036936765s) [noise=true , address=145.79.227.17:4971 , piece_id=3UFDHMLOTFVJFWAW4YMZD3WEWAFP2WVG753HOW5LZCCBJMP3RQQQ, seg_pos={0 0}, upload=494742]: Done uploading piece. err: <nil> Dec 29 03:46:22.167710024 (564h17m22.038793069s) [noise=true , address=145.79.227.48:4301 , piece_id=U7JKK2Z3NA7LKHPMY67NRVTN3LLIONMLQQ7MKPLJKISWWBOF32UQ, seg_pos={0 0}, upload=494742]: Done uploading piece. err: <nil> Dec 29 03:46:22.171214316 (564h17m22.042297351s) [noise=true , address=145.79.227.47:4267 , piece_id=G3ATHA3JGMLQW7GDFAZJNULN35Z2JKHHKSW76Y7ERSY3UHGS3OZQ, seg_pos={0 0}, upload=494742]: Done uploading piece. err: <nil> Dec 29 03:46:22.172234920 (564h17m22.043317955s) [noise=true , address=145.79.227.11:4732 , piece_id=YF7IT7O2RP6BYQCUXDPAEDTES4Y6QGPL2MEWPZUO2EK7Q6BLZAZQ, seg_pos={0 0}, upload=494742]: Done uploading piece. err: <nil> Dec 29 03:46:22.173933193 (564h17m22.045016228s) [noise=true , address=145.79.227.39:4014 , piece_id=ESPZPLPCADOSB7RTOG7FHKOJGSHLXOJ5PLUKJRPJ7XXZR6JZW5MQ, seg_pos={0 0}, upload=494742]: Done uploading piece. err: <nil> Dec 29 03:46:22.347160674 (564h17m22.218243709s) [noise=true , address=145.79.227.32:4835 , piece_id=RLI3XHCNEAJT6NC3AGBPDB2CGNPGOQL4IJCTGP4MFSW37BRQKPPA, seg_pos={0 0}, upload=494742]: Done uploading piece. err: <nil> Dec 29 03:46:22.523668119 (564h17m22.394751164s) [noise=true , address=45.91.116.4:4937 , piece_id=OCSN7SG4ET6EEVKNYJZZ3COK36Z6FGGK5UBFR64ZLUJHOZUTQQKA, seg_pos={0 0}, upload=494742]: Done uploading piece. err: <nil> Dec 29 03:46:22.548878383 (564h17m22.419961418s) [noise=true , address=45.91.116.14:4192 , piece_id=UTIIPUBHYRBXHKPJSAFKS5ZO6SMHA776ZJNL4BWPBAAUNMZTBSAA, seg_pos={0 0}, upload=494742]: Done uploading piece. err: <nil> Dec 29 03:46:22.577987642 (564h17m22.449070677s) [noise=true , address=45.91.116.11:4117 , piece_id=DOG2IOG5JXUMB4EQB4L2VUGHOUPE7623MSPST52XVTBKXBQUERXA, seg_pos={0 0}, upload=494742]: Done uploading piece. err: <nil> Dec 29 03:46:22.778288333 (564h17m22.649371368s) [noise=true , address=45.91.116.13:4156 , piece_id=6PZJDOZHBMD75ISQNWXBFXCIZAN7MEUM62BCRXX6PHJJBXS7BASA, seg_pos={0 0}, upload=494742]: Done uploading piece. err: <nil> Dec 29 03:46:23.343786201 (564h17m23.214869246s) [noise=true , address=45.91.116.7:4010 , piece_id=EQGTWFZXAUEWVCABN6F7MM2GVANVSLREDBR2I5NBTA6YGHZPTUSQ, seg_pos={0 0}, upload=494742]: Done uploading piece. err: <nil> Dec 29 03:46:23.343810318 (564h17m23.214893353s) [seg_pos={0 0}, upload=494742]: Segment reached optimal threshold of 13 pieces. Dec 29 03:46:23.344039785 (564h17m23.21512283s) [noise=true , address=77.90.31.11:4554 , piece_id=2JKGCPJ7XOCPBAYHL6EKTJ5XQGSZQ66JSJBOBICMFU6TQ46RUX4Q, seg_pos={0 0}, upload=494742]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: expected piece hash; context canceled; EOF Dec 29 03:46:23.344049593 (564h17m23.215132658s) [noise=true , address=77.90.31.11:4554 , piece_id=2JKGCPJ7XOCPBAYHL6EKTJ5XQGSZQ66JSJBOBICMFU6TQ46RUX4Q, seg_pos={0 0}, upload=494742]: Long tail cancellation due to optimal threshold Dec 29 03:46:23.344275513 (564h17m23.215358558s) [noise=true , address=77.90.31.17:4773 , piece_id=GVJT4ZD3WZH5PVUBRS7CIJAIOXL27Y4ZOSHYB2MUIGDUQYJERGRQ, seg_pos={0 0}, upload=494742]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ): protocol: write tcp 188.166.225.170:40866->77.90.31.17:4773: use of closed network connection; write tcp 188.166.225.170:40866->77.90.31.17:4773: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40866->77.90.31.17:4773: use of closed network connection Dec 29 03:46:23.344284691 (564h17m23.215367726s) [noise=true , address=77.90.31.17:4773 , piece_id=GVJT4ZD3WZH5PVUBRS7CIJAIOXL27Y4ZOSHYB2MUIGDUQYJERGRQ, seg_pos={0 0}, upload=494742]: Long tail cancellation due to optimal threshold Dec 29 03:46:23.344635579 (564h17m23.215718634s) [noise=true , address=145.79.227.35:4483 , piece_id=W7VC2I6KFENTE24ADU3MG6YZ7WBSM26T255IG4H7SMYULPBNUZHQ, seg_pos={0 0}, upload=494742]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1igSs2ZGX3zWVsSdmm1pnwYd7JckcyBubTuNU1gJDxQsZ9nKaY): protocol: expected piece hash; context canceled; EOF Dec 29 03:46:23.344656268 (564h17m23.215739313s) [noise=true , address=145.79.227.35:4483 , piece_id=W7VC2I6KFENTE24ADU3MG6YZ7WBSM26T255IG4H7SMYULPBNUZHQ, seg_pos={0 0}, upload=494742]: Long tail cancellation due to optimal threshold Dec 29 03:46:23.344695913 (564h17m23.215778948s) [seg_pos={0 0}, upload=494742]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 03:46:23.344816203 (564h17m23.215899238s) [upload=494742]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 03:46:23.344824178 (564h17m23.215907223s) [upload=494742]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 03:46:23.642920316 (564h17m23.514003351s) [upload=494742]: Done uploading segments... Dec 29 03:46:23.642942768 (564h17m23.514025803s) [upload=494742]: Upload finished. err: <nil> Duration: 30.003045162s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [..............................X...................X.......X.] duration: 42.160020103s trace: https://jaeger.dev.storj.io/trace/342d1739235f8379 FAIL (2025-12-29 03:46:38 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== 4:56.793370158 (564h15m56.664453204s) [noise=true , address=145.79.227.20:4441 , piece_id=RBURT4TK6QNESWLHUBFTSFPSYUSSWN3LOI34VXXUFB3R6OZPYTGQ, seg_pos={0 0}, upload=494721]: Uploading piece... Dec 29 03:44:56.794032329 (564h15m56.665115364s) [noise=true , address=77.90.31.19:4846 , piece_id=3XVXPA7JXJNF4KNCIKMCR25N2JSHNQKWW3ZGWWG5YEW66FGRLSKA, seg_pos={0 0}, upload=494721]: Uploading piece... Dec 29 03:44:56.794411752 (564h15m56.665494787s) [noise=true , address=145.79.227.34:4930 , piece_id=HCWSLF7TGOBO2OZSK6T2RA6J4V73DA4NJ7XQDPQQYWAJQY2VXT5Q, seg_pos={0 0}, upload=494721]: Uploading piece... Dec 29 03:44:56.794790392 (564h15m56.665873427s) [noise=true , address=145.79.227.44:4204 , piece_id=Q4HZ75Y77QYSCE3MCQHNLHR2EQOSF6FORAOE2BR7UO5KIGLJFDEA, seg_pos={0 0}, upload=494721]: Uploading piece... Dec 29 03:44:56.795186417 (564h15m56.666269452s) [noise=true , address=145.79.227.35:4497 , piece_id=DYRKGZTU6FTNHHKXDP4G5G7WCYYB7NQN3A6FLZOUK5MUDWBFCEGQ, seg_pos={0 0}, upload=494721]: Uploading piece... Dec 29 03:44:56.795508720 (564h15m56.666591755s) [noise=true , address=145.79.227.40:4066 , piece_id=HYSJ2TRDA4JDP27E7KQCM74KUGJEWMMMO7CPALL6FPGJ6EPG4FKA, seg_pos={0 0}, upload=494721]: Uploading piece... Dec 29 03:44:56.796866796 (564h15m56.667949841s) [noise=true , address=77.90.31.11:4571 , piece_id=YYZLQ7ICV7ELGHIZ22YEQLHW5GW7ZX6BRORS7MDWPPGQZR3XZCWA, seg_pos={0 0}, upload=494721]: Uploading piece... Dec 29 03:44:56.798305896 (564h15m56.669388931s) [noise=true , address=145.79.227.7:4326 , piece_id=UFIGP5Q3XYHQYVBV6NAVZRKMP3TNL2CK7TW25EX55KVIH56ABFOQ, seg_pos={0 0}, upload=494721]: Uploading piece... Dec 29 03:44:56.800160058 (564h15m56.671243093s) [noise=true , address=77.90.31.16:4740 , piece_id=BVQGUHICWNPAOAEONW4D2O7RTNQG46TPRL62X6PZPSM2DPKSEXOQ, seg_pos={0 0}, upload=494721]: Uploading piece... Dec 29 03:44:56.801740887 (564h15m56.672823932s) [noise=true , address=45.91.116.6:4003 , piece_id=NXZ2IO7VXPKICQ5Q7F2KRFL43BOGM6OEFMPESRXF3Q3Z7HC4YBVA, seg_pos={0 0}, upload=494721]: Uploading piece... Dec 29 03:44:56.804036919 (564h15m56.675119964s) [noise=true , address=145.79.227.29:4958 , piece_id=QJZXUAR2V537ILKBYM67C2GISYWH42R7PZSIKGYMI2GN72ZHONTQ, seg_pos={0 0}, upload=494721]: Uploading piece... Dec 29 03:44:56.807028817 (564h15m56.678111862s) [noise=true , address=145.79.227.9:4395 , piece_id=AFQI35LDQ5LDGJCQULPJVKJXOFFMHM4IXYA47X72PDZQ72BZYDWQ, seg_pos={0 0}, upload=494721]: Uploading piece... Dec 29 03:44:56.808482935 (564h15m56.67956597s) [noise=true , address=77.90.31.14:4685 , piece_id=PKWRWGS37YE7AZI2AOBC5EJNMMC6MZTNNMV2ZCEE552QUJR3NXHA, seg_pos={0 0}, upload=494721]: Uploading piece... Dec 29 03:44:56.809876388 (564h15m56.680959423s) [noise=true , address=145.79.227.33:4882 , piece_id=D7M67HYADXEQKVO3ADWTKFHD3EKYZC2UWG2EELPZWLR4NTMMNKRQ, seg_pos={0 0}, upload=494721]: Uploading piece... Dec 29 03:44:56.812047603 (564h15m56.683130638s) [noise=true , address=145.79.227.30:4767 , piece_id=N7HS726RWAMRYSILZBYU7RZFZ3ME4ALCD2D6CCBHBC4WHZ5SQGBQ, seg_pos={0 0}, upload=494721]: Uploading piece... Dec 29 03:44:56.812335190 (564h15m56.683418225s) [noise=true , address=77.90.31.16:4737 , piece_id=2ENSKEQAIUJDDB2YG6SZIQD4KMPZL5UFPV3YTSCH4BMNB5L3CYEA, seg_pos={0 0}, upload=494721]: Uploading piece... Dec 29 03:44:56.813654823 (564h15m56.684737868s) [noise=true , address=145.79.227.49:4359 , piece_id=QFNQZ3OFSRDD3GKWMAJBXMX5CPRPLC2D4T4JUFTWYGPYBAM5GTQA, seg_pos={0 0}, upload=494721]: Uploading piece... Dec 29 03:44:58.252946862 (564h15m58.124029907s) [upload=494721]: Next returned nil segment Dec 29 03:44:58.252960116 (564h15m58.124043162s) [upload=494721]: Waiting for error group managing segments... Dec 29 03:44:58.645366725 (564h15m58.51644977s) [noise=true , address=145.79.227.20:4441 , piece_id=RBURT4TK6QNESWLHUBFTSFPSYUSSWN3LOI34VXXUFB3R6OZPYTGQ, seg_pos={0 0}, upload=494721]: Done uploading piece. err: <nil> Dec 29 03:44:58.670319589 (564h15m58.541402644s) [noise=true , address=145.79.227.9:4395 , piece_id=AFQI35LDQ5LDGJCQULPJVKJXOFFMHM4IXYA47X72PDZQ72BZYDWQ, seg_pos={0 0}, upload=494721]: Done uploading piece. err: <nil> Dec 29 03:44:58.675517227 (564h15m58.546600262s) [noise=true , address=145.79.227.7:4326 , piece_id=UFIGP5Q3XYHQYVBV6NAVZRKMP3TNL2CK7TW25EX55KVIH56ABFOQ, seg_pos={0 0}, upload=494721]: Done uploading piece. err: <nil> Dec 29 03:44:58.681999799 (564h15m58.553082834s) [noise=true , address=145.79.227.12:4771 , piece_id=TWG66VNCPR6PIZHQXLEJMO3EROZWCBJBX3IQ3AHL5JOQQMU7H34Q, seg_pos={0 0}, upload=494721]: Done uploading piece. err: <nil> Dec 29 03:44:58.707798995 (564h15m58.57888203s) [noise=true , address=145.79.227.29:4958 , piece_id=QJZXUAR2V537ILKBYM67C2GISYWH42R7PZSIKGYMI2GN72ZHONTQ, seg_pos={0 0}, upload=494721]: Done uploading piece. err: <nil> Dec 29 03:44:58.883206445 (564h15m58.75428948s) [noise=true , address=77.90.31.16:4737 , piece_id=2ENSKEQAIUJDDB2YG6SZIQD4KMPZL5UFPV3YTSCH4BMNB5L3CYEA, seg_pos={0 0}, upload=494721]: Done uploading piece. err: <nil> Dec 29 03:44:58.901139292 (564h15m58.772222328s) [noise=true , address=45.91.116.6:4003 , piece_id=NXZ2IO7VXPKICQ5Q7F2KRFL43BOGM6OEFMPESRXF3Q3Z7HC4YBVA, seg_pos={0 0}, upload=494721]: Done uploading piece. err: <nil> Dec 29 03:44:58.935456668 (564h15m58.806539703s) [noise=true , address=77.90.31.19:4846 , piece_id=3XVXPA7JXJNF4KNCIKMCR25N2JSHNQKWW3ZGWWG5YEW66FGRLSKA, seg_pos={0 0}, upload=494721]: Done uploading piece. err: <nil> Dec 29 03:44:59.034389139 (564h15m58.905472174s) [noise=true , address=77.90.31.11:4571 , piece_id=YYZLQ7ICV7ELGHIZ22YEQLHW5GW7ZX6BRORS7MDWPPGQZR3XZCWA, seg_pos={0 0}, upload=494721]: Done uploading piece. err: <nil> Dec 29 03:44:59.083013598 (564h15m58.954096633s) [noise=true , address=77.90.31.14:4685 , piece_id=PKWRWGS37YE7AZI2AOBC5EJNMMC6MZTNNMV2ZCEE552QUJR3NXHA, seg_pos={0 0}, upload=494721]: Done uploading piece. err: <nil> Dec 29 03:44:59.577054311 (564h15m59.448137346s) [noise=true , address=145.79.227.44:4204 , piece_id=Q4HZ75Y77QYSCE3MCQHNLHR2EQOSF6FORAOE2BR7UO5KIGLJFDEA, seg_pos={0 0}, upload=494721]: Done uploading piece. err: <nil> Dec 29 03:44:59.599698558 (564h15m59.470781593s) [noise=true , address=145.79.227.30:4767 , piece_id=N7HS726RWAMRYSILZBYU7RZFZ3ME4ALCD2D6CCBHBC4WHZ5SQGBQ, seg_pos={0 0}, upload=494721]: Done uploading piece. err: <nil> Dec 29 03:44:59.615985350 (564h15m59.487068385s) [noise=true , address=145.79.227.35:4497 , piece_id=DYRKGZTU6FTNHHKXDP4G5G7WCYYB7NQN3A6FLZOUK5MUDWBFCEGQ, seg_pos={0 0}, upload=494721]: Done uploading piece. err: <nil> Dec 29 03:44:59.649599526 (564h15m59.520682561s) [noise=true , address=145.79.227.33:4882 , piece_id=D7M67HYADXEQKVO3ADWTKFHD3EKYZC2UWG2EELPZWLR4NTMMNKRQ, seg_pos={0 0}, upload=494721]: Done uploading piece. err: <nil> Dec 29 03:44:59.649617401 (564h15m59.520700436s) [seg_pos={0 0}, upload=494721]: Segment reached optimal threshold of 14 pieces. Dec 29 03:44:59.649785541 (564h15m59.520868576s) [noise=true , address=145.79.227.40:4066 , piece_id=HYSJ2TRDA4JDP27E7KQCM74KUGJEWMMMO7CPALL6FPGJ6EPG4FKA, seg_pos={0 0}, upload=494721]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tRYJFF6VmxGk8nr1nh6UD3Fnq76qu2Tgkq7cWpV6E8GVpsAUC): protocol: expected piece hash; context canceled; EOF Dec 29 03:44:59.649793366 (564h15m59.520876401s) [noise=true , address=145.79.227.40:4066 , piece_id=HYSJ2TRDA4JDP27E7KQCM74KUGJEWMMMO7CPALL6FPGJ6EPG4FKA, seg_pos={0 0}, upload=494721]: Long tail cancellation due to optimal threshold Dec 29 03:44:59.649844793 (564h15m59.520927828s) [noise=true , address=145.79.227.49:4359 , piece_id=QFNQZ3OFSRDD3GKWMAJBXMX5CPRPLC2D4T4JUFTWYGPYBAM5GTQA, seg_pos={0 0}, upload=494721]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YgFc7P9RCbYw9u2wfow8Z3DMwFvJ6AnCsvzoxU9pTwPWGw8MV): protocol: expected piece hash; context canceled; EOF Dec 29 03:44:59.649857167 (564h15m59.520940202s) [noise=true , address=145.79.227.49:4359 , piece_id=QFNQZ3OFSRDD3GKWMAJBXMX5CPRPLC2D4T4JUFTWYGPYBAM5GTQA, seg_pos={0 0}, upload=494721]: Long tail cancellation due to optimal threshold Dec 29 03:44:59.649989739 (564h15m59.521072774s) [noise=true , address=77.90.31.16:4740 , piece_id=BVQGUHICWNPAOAEONW4D2O7RTNQG46TPRL62X6PZPSM2DPKSEXOQ, seg_pos={0 0}, upload=494721]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:145raYQJozNw3SMbj78wDsR6EjPvJTqqcocf1SaPs3HEnNVS5p): protocol: expected piece hash; context canceled; EOF Dec 29 03:44:59.649995230 (564h15m59.521078265s) [noise=true , address=77.90.31.16:4740 , piece_id=BVQGUHICWNPAOAEONW4D2O7RTNQG46TPRL62X6PZPSM2DPKSEXOQ, seg_pos={0 0}, upload=494721]: Long tail cancellation due to optimal threshold Dec 29 03:44:59.650065975 (564h15m59.52114901s) [noise=true , address=145.79.227.34:4930 , piece_id=HCWSLF7TGOBO2OZSK6T2RA6J4V73DA4NJ7XQDPQQYWAJQY2VXT5Q, seg_pos={0 0}, upload=494721]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qaYFSpezuLAc5zJfAYruzad6nTpYFDpog7bmKMVTA8MVm76NB): protocol: expected piece hash; context canceled; EOF Dec 29 03:44:59.650072357 (564h15m59.521155392s) [noise=true , address=145.79.227.34:4930 , piece_id=HCWSLF7TGOBO2OZSK6T2RA6J4V73DA4NJ7XQDPQQYWAJQY2VXT5Q, seg_pos={0 0}, upload=494721]: Long tail cancellation due to optimal threshold Dec 29 03:44:59.650100330 (564h15m59.521183365s) [seg_pos={0 0}, upload=494721]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 03:44:59.650189079 (564h15m59.521272114s) [upload=494721]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 03:44:59.650193397 (564h15m59.521276432s) [upload=494721]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 03:44:59.950568254 (564h15m59.821651289s) [upload=494721]: Done uploading segments... Dec 29 03:44:59.950645721 (564h15m59.821728766s) [upload=494721]: Upload finished. err: <nil> Duration: 42.160037856s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 7 out of 60 is failed (threshold=6) [.........................XX................X......XXX......X] duration: 5.001782493s trace: https://jaeger.dev.storj.io/trace/f91d358bd5ae577 FAIL (2025-12-29 03:46:13 UTC) orbiter warning: context deadline exceeded Duration: 5.001806748s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [...............................X...................X.......X] duration: 1m0.003017593s trace: https://jaeger.dev.storj.io/trace/342d1739235f8379 FAIL (2025-12-29 03:45:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== 4:56.793370158 (564h15m56.664453204s) [noise=true , address=145.79.227.20:4441 , piece_id=RBURT4TK6QNESWLHUBFTSFPSYUSSWN3LOI34VXXUFB3R6OZPYTGQ, seg_pos={0 0}, upload=494721]: Uploading piece... Dec 29 03:44:56.794032329 (564h15m56.665115364s) [noise=true , address=77.90.31.19:4846 , piece_id=3XVXPA7JXJNF4KNCIKMCR25N2JSHNQKWW3ZGWWG5YEW66FGRLSKA, seg_pos={0 0}, upload=494721]: Uploading piece... Dec 29 03:44:56.794411752 (564h15m56.665494787s) [noise=true , address=145.79.227.34:4930 , piece_id=HCWSLF7TGOBO2OZSK6T2RA6J4V73DA4NJ7XQDPQQYWAJQY2VXT5Q, seg_pos={0 0}, upload=494721]: Uploading piece... Dec 29 03:44:56.794790392 (564h15m56.665873427s) [noise=true , address=145.79.227.44:4204 , piece_id=Q4HZ75Y77QYSCE3MCQHNLHR2EQOSF6FORAOE2BR7UO5KIGLJFDEA, seg_pos={0 0}, upload=494721]: Uploading piece... Dec 29 03:44:56.795186417 (564h15m56.666269452s) [noise=true , address=145.79.227.35:4497 , piece_id=DYRKGZTU6FTNHHKXDP4G5G7WCYYB7NQN3A6FLZOUK5MUDWBFCEGQ, seg_pos={0 0}, upload=494721]: Uploading piece... Dec 29 03:44:56.795508720 (564h15m56.666591755s) [noise=true , address=145.79.227.40:4066 , piece_id=HYSJ2TRDA4JDP27E7KQCM74KUGJEWMMMO7CPALL6FPGJ6EPG4FKA, seg_pos={0 0}, upload=494721]: Uploading piece... Dec 29 03:44:56.796866796 (564h15m56.667949841s) [noise=true , address=77.90.31.11:4571 , piece_id=YYZLQ7ICV7ELGHIZ22YEQLHW5GW7ZX6BRORS7MDWPPGQZR3XZCWA, seg_pos={0 0}, upload=494721]: Uploading piece... Dec 29 03:44:56.798305896 (564h15m56.669388931s) [noise=true , address=145.79.227.7:4326 , piece_id=UFIGP5Q3XYHQYVBV6NAVZRKMP3TNL2CK7TW25EX55KVIH56ABFOQ, seg_pos={0 0}, upload=494721]: Uploading piece... Dec 29 03:44:56.800160058 (564h15m56.671243093s) [noise=true , address=77.90.31.16:4740 , piece_id=BVQGUHICWNPAOAEONW4D2O7RTNQG46TPRL62X6PZPSM2DPKSEXOQ, seg_pos={0 0}, upload=494721]: Uploading piece... Dec 29 03:44:56.801740887 (564h15m56.672823932s) [noise=true , address=45.91.116.6:4003 , piece_id=NXZ2IO7VXPKICQ5Q7F2KRFL43BOGM6OEFMPESRXF3Q3Z7HC4YBVA, seg_pos={0 0}, upload=494721]: Uploading piece... Dec 29 03:44:56.804036919 (564h15m56.675119964s) [noise=true , address=145.79.227.29:4958 , piece_id=QJZXUAR2V537ILKBYM67C2GISYWH42R7PZSIKGYMI2GN72ZHONTQ, seg_pos={0 0}, upload=494721]: Uploading piece... Dec 29 03:44:56.807028817 (564h15m56.678111862s) [noise=true , address=145.79.227.9:4395 , piece_id=AFQI35LDQ5LDGJCQULPJVKJXOFFMHM4IXYA47X72PDZQ72BZYDWQ, seg_pos={0 0}, upload=494721]: Uploading piece... Dec 29 03:44:56.808482935 (564h15m56.67956597s) [noise=true , address=77.90.31.14:4685 , piece_id=PKWRWGS37YE7AZI2AOBC5EJNMMC6MZTNNMV2ZCEE552QUJR3NXHA, seg_pos={0 0}, upload=494721]: Uploading piece... Dec 29 03:44:56.809876388 (564h15m56.680959423s) [noise=true , address=145.79.227.33:4882 , piece_id=D7M67HYADXEQKVO3ADWTKFHD3EKYZC2UWG2EELPZWLR4NTMMNKRQ, seg_pos={0 0}, upload=494721]: Uploading piece... Dec 29 03:44:56.812047603 (564h15m56.683130638s) [noise=true , address=145.79.227.30:4767 , piece_id=N7HS726RWAMRYSILZBYU7RZFZ3ME4ALCD2D6CCBHBC4WHZ5SQGBQ, seg_pos={0 0}, upload=494721]: Uploading piece... Dec 29 03:44:56.812335190 (564h15m56.683418225s) [noise=true , address=77.90.31.16:4737 , piece_id=2ENSKEQAIUJDDB2YG6SZIQD4KMPZL5UFPV3YTSCH4BMNB5L3CYEA, seg_pos={0 0}, upload=494721]: Uploading piece... Dec 29 03:44:56.813654823 (564h15m56.684737868s) [noise=true , address=145.79.227.49:4359 , piece_id=QFNQZ3OFSRDD3GKWMAJBXMX5CPRPLC2D4T4JUFTWYGPYBAM5GTQA, seg_pos={0 0}, upload=494721]: Uploading piece... Dec 29 03:44:58.252946862 (564h15m58.124029907s) [upload=494721]: Next returned nil segment Dec 29 03:44:58.252960116 (564h15m58.124043162s) [upload=494721]: Waiting for error group managing segments... Dec 29 03:44:58.645366725 (564h15m58.51644977s) [noise=true , address=145.79.227.20:4441 , piece_id=RBURT4TK6QNESWLHUBFTSFPSYUSSWN3LOI34VXXUFB3R6OZPYTGQ, seg_pos={0 0}, upload=494721]: Done uploading piece. err: <nil> Dec 29 03:44:58.670319589 (564h15m58.541402644s) [noise=true , address=145.79.227.9:4395 , piece_id=AFQI35LDQ5LDGJCQULPJVKJXOFFMHM4IXYA47X72PDZQ72BZYDWQ, seg_pos={0 0}, upload=494721]: Done uploading piece. err: <nil> Dec 29 03:44:58.675517227 (564h15m58.546600262s) [noise=true , address=145.79.227.7:4326 , piece_id=UFIGP5Q3XYHQYVBV6NAVZRKMP3TNL2CK7TW25EX55KVIH56ABFOQ, seg_pos={0 0}, upload=494721]: Done uploading piece. err: <nil> Dec 29 03:44:58.681999799 (564h15m58.553082834s) [noise=true , address=145.79.227.12:4771 , piece_id=TWG66VNCPR6PIZHQXLEJMO3EROZWCBJBX3IQ3AHL5JOQQMU7H34Q, seg_pos={0 0}, upload=494721]: Done uploading piece. err: <nil> Dec 29 03:44:58.707798995 (564h15m58.57888203s) [noise=true , address=145.79.227.29:4958 , piece_id=QJZXUAR2V537ILKBYM67C2GISYWH42R7PZSIKGYMI2GN72ZHONTQ, seg_pos={0 0}, upload=494721]: Done uploading piece. err: <nil> Dec 29 03:44:58.883206445 (564h15m58.75428948s) [noise=true , address=77.90.31.16:4737 , piece_id=2ENSKEQAIUJDDB2YG6SZIQD4KMPZL5UFPV3YTSCH4BMNB5L3CYEA, seg_pos={0 0}, upload=494721]: Done uploading piece. err: <nil> Dec 29 03:44:58.901139292 (564h15m58.772222328s) [noise=true , address=45.91.116.6:4003 , piece_id=NXZ2IO7VXPKICQ5Q7F2KRFL43BOGM6OEFMPESRXF3Q3Z7HC4YBVA, seg_pos={0 0}, upload=494721]: Done uploading piece. err: <nil> Dec 29 03:44:58.935456668 (564h15m58.806539703s) [noise=true , address=77.90.31.19:4846 , piece_id=3XVXPA7JXJNF4KNCIKMCR25N2JSHNQKWW3ZGWWG5YEW66FGRLSKA, seg_pos={0 0}, upload=494721]: Done uploading piece. err: <nil> Dec 29 03:44:59.034389139 (564h15m58.905472174s) [noise=true , address=77.90.31.11:4571 , piece_id=YYZLQ7ICV7ELGHIZ22YEQLHW5GW7ZX6BRORS7MDWPPGQZR3XZCWA, seg_pos={0 0}, upload=494721]: Done uploading piece. err: <nil> Dec 29 03:44:59.083013598 (564h15m58.954096633s) [noise=true , address=77.90.31.14:4685 , piece_id=PKWRWGS37YE7AZI2AOBC5EJNMMC6MZTNNMV2ZCEE552QUJR3NXHA, seg_pos={0 0}, upload=494721]: Done uploading piece. err: <nil> Dec 29 03:44:59.577054311 (564h15m59.448137346s) [noise=true , address=145.79.227.44:4204 , piece_id=Q4HZ75Y77QYSCE3MCQHNLHR2EQOSF6FORAOE2BR7UO5KIGLJFDEA, seg_pos={0 0}, upload=494721]: Done uploading piece. err: <nil> Dec 29 03:44:59.599698558 (564h15m59.470781593s) [noise=true , address=145.79.227.30:4767 , piece_id=N7HS726RWAMRYSILZBYU7RZFZ3ME4ALCD2D6CCBHBC4WHZ5SQGBQ, seg_pos={0 0}, upload=494721]: Done uploading piece. err: <nil> Dec 29 03:44:59.615985350 (564h15m59.487068385s) [noise=true , address=145.79.227.35:4497 , piece_id=DYRKGZTU6FTNHHKXDP4G5G7WCYYB7NQN3A6FLZOUK5MUDWBFCEGQ, seg_pos={0 0}, upload=494721]: Done uploading piece. err: <nil> Dec 29 03:44:59.649599526 (564h15m59.520682561s) [noise=true , address=145.79.227.33:4882 , piece_id=D7M67HYADXEQKVO3ADWTKFHD3EKYZC2UWG2EELPZWLR4NTMMNKRQ, seg_pos={0 0}, upload=494721]: Done uploading piece. err: <nil> Dec 29 03:44:59.649617401 (564h15m59.520700436s) [seg_pos={0 0}, upload=494721]: Segment reached optimal threshold of 14 pieces. Dec 29 03:44:59.649785541 (564h15m59.520868576s) [noise=true , address=145.79.227.40:4066 , piece_id=HYSJ2TRDA4JDP27E7KQCM74KUGJEWMMMO7CPALL6FPGJ6EPG4FKA, seg_pos={0 0}, upload=494721]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tRYJFF6VmxGk8nr1nh6UD3Fnq76qu2Tgkq7cWpV6E8GVpsAUC): protocol: expected piece hash; context canceled; EOF Dec 29 03:44:59.649793366 (564h15m59.520876401s) [noise=true , address=145.79.227.40:4066 , piece_id=HYSJ2TRDA4JDP27E7KQCM74KUGJEWMMMO7CPALL6FPGJ6EPG4FKA, seg_pos={0 0}, upload=494721]: Long tail cancellation due to optimal threshold Dec 29 03:44:59.649844793 (564h15m59.520927828s) [noise=true , address=145.79.227.49:4359 , piece_id=QFNQZ3OFSRDD3GKWMAJBXMX5CPRPLC2D4T4JUFTWYGPYBAM5GTQA, seg_pos={0 0}, upload=494721]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YgFc7P9RCbYw9u2wfow8Z3DMwFvJ6AnCsvzoxU9pTwPWGw8MV): protocol: expected piece hash; context canceled; EOF Dec 29 03:44:59.649857167 (564h15m59.520940202s) [noise=true , address=145.79.227.49:4359 , piece_id=QFNQZ3OFSRDD3GKWMAJBXMX5CPRPLC2D4T4JUFTWYGPYBAM5GTQA, seg_pos={0 0}, upload=494721]: Long tail cancellation due to optimal threshold Dec 29 03:44:59.649989739 (564h15m59.521072774s) [noise=true , address=77.90.31.16:4740 , piece_id=BVQGUHICWNPAOAEONW4D2O7RTNQG46TPRL62X6PZPSM2DPKSEXOQ, seg_pos={0 0}, upload=494721]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:145raYQJozNw3SMbj78wDsR6EjPvJTqqcocf1SaPs3HEnNVS5p): protocol: expected piece hash; context canceled; EOF Dec 29 03:44:59.649995230 (564h15m59.521078265s) [noise=true , address=77.90.31.16:4740 , piece_id=BVQGUHICWNPAOAEONW4D2O7RTNQG46TPRL62X6PZPSM2DPKSEXOQ, seg_pos={0 0}, upload=494721]: Long tail cancellation due to optimal threshold Dec 29 03:44:59.650065975 (564h15m59.52114901s) [noise=true , address=145.79.227.34:4930 , piece_id=HCWSLF7TGOBO2OZSK6T2RA6J4V73DA4NJ7XQDPQQYWAJQY2VXT5Q, seg_pos={0 0}, upload=494721]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qaYFSpezuLAc5zJfAYruzad6nTpYFDpog7bmKMVTA8MVm76NB): protocol: expected piece hash; context canceled; EOF Dec 29 03:44:59.650072357 (564h15m59.521155392s) [noise=true , address=145.79.227.34:4930 , piece_id=HCWSLF7TGOBO2OZSK6T2RA6J4V73DA4NJ7XQDPQQYWAJQY2VXT5Q, seg_pos={0 0}, upload=494721]: Long tail cancellation due to optimal threshold Dec 29 03:44:59.650100330 (564h15m59.521183365s) [seg_pos={0 0}, upload=494721]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 03:44:59.650189079 (564h15m59.521272114s) [upload=494721]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 03:44:59.650193397 (564h15m59.521276432s) [upload=494721]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 03:44:59.950568254 (564h15m59.821651289s) [upload=494721]: Done uploading segments... Dec 29 03:44:59.950645721 (564h15m59.821728766s) [upload=494721]: Upload finished. err: <nil> Duration: 1m0.003030529s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 30 out of 60 is failed (threshold=3) [....XX.X.....X.X..XXXX..XXX.XX...X.X.....XXX...XXXXXXX.XX.XX] duration: 30.003462784s trace: https://jaeger.dev.storj.io/trace/33a0e17fe5b7e602 FAIL (2025-12-29 03:45:42 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, Piece ID: KE2GETMJMXQ2HA34CFQD23ITLVTYVHEE2Q5ISVCCL2CEPKPU5MVQ) context deadline exceeded; piecestore: (Node ID: 1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, Piece ID: 3KTYQXBFLONJUVURY5CQRY5TNHJKUNU4K2HUPAICD22F5E2NKPLQ) context deadline exceeded ======LOGS====== 03:45:13.219045188 (564h16m13.090128233s) [upload=494726]: Starting upload Dec 29 03:45:13.219135590 (564h16m13.090218635s) [upload=494726]: Done starting upload Dec 29 03:45:13.219196346 (564h16m13.090279391s) [upload=494726]: Uploading segments... Dec 29 03:45:13.219393942 (564h16m13.090476987s) [upload=494726]: Got next segment. Inline: false Dec 29 03:45:13.219443507 (564h16m13.090526552s) [upload=494726]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 03:45:13.219450019 (564h16m13.090533064s) [upload=494726]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 03:45:13.460062800 (564h16m13.331145845s) [seg_pos={0 0}, upload=494726]: Begin upload segment... Dec 29 03:45:13.460121743 (564h16m13.331204778s) [seg_pos={0 0}, upload=494726]: Done begin upload segment. Dec 29 03:45:13.460215410 (564h16m13.331298445s) [noise=true , address=145.79.227.51:4411 , piece_id=B6RTE3R65MZE7LX6FO57ANHSYYKICFDINX43WJV6BJZ6DJTUN54Q, seg_pos={0 0}, upload=494726]: Uploading piece... Dec 29 03:45:13.460438205 (564h16m13.33152125s) [noise=true , address=77.90.31.14:4662 , piece_id=ITJMRNF54I7WF6OWA4UWRZ7QXLDB6JTTB5BXVFXY3VSID3YWT5VA, seg_pos={0 0}, upload=494726]: Uploading piece... Dec 29 03:45:13.461008371 (564h16m13.332091416s) [noise=true , address=77.90.31.19:4842 , piece_id=HED4RV5WY5A7XWUVOT3QRUTCTDGP6CTV7FIMKVUYFZJQKBDOB63Q, seg_pos={0 0}, upload=494726]: Uploading piece... Dec 29 03:45:13.461168346 (564h16m13.332251381s) [noise=true , address=145.79.227.15:4876 , piece_id=67DRZISWSKNHEZLHA5XXKEWFI4UH52556OVCUE5NC6DK6UDU7QXA, seg_pos={0 0}, upload=494726]: Uploading piece... Dec 29 03:45:13.461591581 (564h16m13.332674626s) [noise=true , address=45.91.116.11:4118 , piece_id=KE2GETMJMXQ2HA34CFQD23ITLVTYVHEE2Q5ISVCCL2CEPKPU5MVQ, seg_pos={0 0}, upload=494726]: Uploading piece... Dec 29 03:45:13.462036619 (564h16m13.333119664s) [noise=true , address=77.90.31.21:4880 , piece_id=QZQTKPTOOYXPWES4XIJ453JBDAEK4APUVRXOBTU3DZBFX44C2MDQ, seg_pos={0 0}, upload=494726]: Uploading piece... Dec 29 03:45:13.462086654 (564h16m13.333169689s) [noise=true , address=145.79.227.52:4447 , piece_id=UJ3WPEXGV7K2FLGBVRVPDNJHW7P36TSXNVRLDPMHVSOE4XYWJ44Q, seg_pos={0 0}, upload=494726]: Uploading piece... Dec 29 03:45:13.462422654 (564h16m13.333505689s) [noise=true , address=145.79.227.26:4653 , piece_id=WHWBAZBL5MYOKS2A4LNRQI4NM362LMCG6YNM6S5EBV64L7QMNP2Q, seg_pos={0 0}, upload=494726]: Uploading piece... Dec 29 03:45:13.462478400 (564h16m13.333561435s) [noise=true , address=145.79.227.14:4837 , piece_id=DR7EVP5IKVJC7OICVLN7SUPDDLGBKJKNPZIQHWVQRCDXAB2BM3YQ, seg_pos={0 0}, upload=494726]: Uploading piece... Dec 29 03:45:13.462794041 (564h16m13.333877076s) [noise=true , address=145.79.227.11:4731 , piece_id=CPT24BPRESFN7KHHTOYWKE6D3PXV7OTY46ENDVRZVCU6WPCIFRIA, seg_pos={0 0}, upload=494726]: Uploading piece... Dec 29 03:45:13.463004151 (564h16m13.334087196s) [noise=true , address=45.91.116.13:4156 , piece_id=3KTYQXBFLONJUVURY5CQRY5TNHJKUNU4K2HUPAICD22F5E2NKPLQ, seg_pos={0 0}, upload=494726]: Uploading piece... Dec 29 03:45:13.463120713 (564h16m13.334203748s) [noise=true , address=145.79.227.39:4015 , piece_id=QQHLIUAHG2IQSABDRCOTXVFFSWVTA4DN375VVB4ULJZBNHZDOYVA, seg_pos={0 0}, upload=494726]: Uploading piece... Dec 29 03:45:13.463397550 (564h16m13.334480595s) [noise=true , address=145.79.227.36:4519 , piece_id=BN6SE3RTAT5R7QTYKPC24KCLZZAC3LJ4X7JYPCQ2C2HOUWDPHNOA, seg_pos={0 0}, upload=494726]: Uploading piece... Dec 29 03:45:13.463425704 (564h16m13.334508749s) [noise=true , address=45.91.116.10:4082 , piece_id=H7ZX44Z6OJOY2QSWPRMIXNOW42ZKQHZP7BX24XEX7P7MW5V5SNMA, seg_pos={0 0}, upload=494726]: Uploading piece... Dec 29 03:45:13.463832348 (564h16m13.334915393s) [noise=true , address=45.91.116.14:4191 , piece_id=BMWVFCCTAKVWFA5XDGAS34QOTGQJWKVQL4J4EV4GKCFCY6DZW4DQ, seg_pos={0 0}, upload=494726]: Uploading piece... Dec 29 03:45:13.464151105 (564h16m13.33523414s) [noise=true , address=77.90.31.13:4629 , piece_id=WON727FA2R2B5C2NEN4VMFIS76UD7TFTRAO5XJJCDYCE4MAFIVBA, seg_pos={0 0}, upload=494726]: Uploading piece... Dec 29 03:45:15.159805884 (564h16m15.03088893s) [upload=494726]: Next returned nil segment Dec 29 03:45:15.159831243 (564h16m15.030914288s) [upload=494726]: Waiting for error group managing segments... Dec 29 03:45:15.444249656 (564h16m15.315332691s) [noise=true , address=145.79.227.11:4731 , piece_id=CPT24BPRESFN7KHHTOYWKE6D3PXV7OTY46ENDVRZVCU6WPCIFRIA, seg_pos={0 0}, upload=494726]: Done uploading piece. err: <nil> Dec 29 03:45:15.446281765 (564h16m15.317364801s) [noise=true , address=145.79.227.39:4015 , piece_id=QQHLIUAHG2IQSABDRCOTXVFFSWVTA4DN375VVB4ULJZBNHZDOYVA, seg_pos={0 0}, upload=494726]: Done uploading piece. err: <nil> Dec 29 03:45:15.447576000 (564h16m15.318659035s) [noise=true , address=145.79.227.14:4837 , piece_id=DR7EVP5IKVJC7OICVLN7SUPDDLGBKJKNPZIQHWVQRCDXAB2BM3YQ, seg_pos={0 0}, upload=494726]: Done uploading piece. err: <nil> Dec 29 03:45:15.449977903 (564h16m15.321060938s) [noise=true , address=145.79.227.52:4447 , piece_id=UJ3WPEXGV7K2FLGBVRVPDNJHW7P36TSXNVRLDPMHVSOE4XYWJ44Q, seg_pos={0 0}, upload=494726]: Done uploading piece. err: <nil> Dec 29 03:45:15.459196117 (564h16m15.330279152s) [noise=true , address=145.79.227.51:4411 , piece_id=B6RTE3R65MZE7LX6FO57ANHSYYKICFDINX43WJV6BJZ6DJTUN54Q, seg_pos={0 0}, upload=494726]: Done uploading piece. err: <nil> Dec 29 03:45:15.465256376 (564h16m15.336339421s) [noise=true , address=145.79.227.15:4876 , piece_id=67DRZISWSKNHEZLHA5XXKEWFI4UH52556OVCUE5NC6DK6UDU7QXA, seg_pos={0 0}, upload=494726]: Done uploading piece. err: <nil> Dec 29 03:45:15.472904739 (564h16m15.343987785s) [noise=true , address=145.79.227.26:4653 , piece_id=WHWBAZBL5MYOKS2A4LNRQI4NM362LMCG6YNM6S5EBV64L7QMNP2Q, seg_pos={0 0}, upload=494726]: Done uploading piece. err: <nil> Dec 29 03:45:15.475900674 (564h16m15.346983709s) [noise=true , address=145.79.227.36:4519 , piece_id=BN6SE3RTAT5R7QTYKPC24KCLZZAC3LJ4X7JYPCQ2C2HOUWDPHNOA, seg_pos={0 0}, upload=494726]: Done uploading piece. err: <nil> Dec 29 03:45:15.670275473 (564h16m15.541358518s) [noise=true , address=77.90.31.21:4880 , piece_id=QZQTKPTOOYXPWES4XIJ453JBDAEK4APUVRXOBTU3DZBFX44C2MDQ, seg_pos={0 0}, upload=494726]: Done uploading piece. err: <nil> Dec 29 03:45:15.706164480 (564h16m15.577247516s) [noise=true , address=45.91.116.10:4082 , piece_id=H7ZX44Z6OJOY2QSWPRMIXNOW42ZKQHZP7BX24XEX7P7MW5V5SNMA, seg_pos={0 0}, upload=494726]: Done uploading piece. err: <nil> Dec 29 03:45:15.719382830 (564h16m15.590465865s) [noise=true , address=45.91.116.13:4156 , piece_id=3KTYQXBFLONJUVURY5CQRY5TNHJKUNU4K2HUPAICD22F5E2NKPLQ, seg_pos={0 0}, upload=494726]: Done uploading piece. err: <nil> Dec 29 03:45:15.748767923 (564h16m15.619850968s) [noise=true , address=77.90.31.19:4842 , piece_id=HED4RV5WY5A7XWUVOT3QRUTCTDGP6CTV7FIMKVUYFZJQKBDOB63Q, seg_pos={0 0}, upload=494726]: Done uploading piece. err: <nil> Dec 29 03:45:15.755768613 (564h16m15.626851658s) [noise=true , address=45.91.116.11:4118 , piece_id=KE2GETMJMXQ2HA34CFQD23ITLVTYVHEE2Q5ISVCCL2CEPKPU5MVQ, seg_pos={0 0}, upload=494726]: Done uploading piece. err: <nil> Dec 29 03:45:15.755792910 (564h16m15.626875945s) [seg_pos={0 0}, upload=494726]: Segment reached optimal threshold of 13 pieces. Dec 29 03:45:15.756054257 (564h16m15.627137302s) [noise=true , address=77.90.31.13:4629 , piece_id=WON727FA2R2B5C2NEN4VMFIS76UD7TFTRAO5XJJCDYCE4MAFIVBA, seg_pos={0 0}, upload=494726]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f): protocol: expected piece hash; context canceled; EOF Dec 29 03:45:15.756063034 (564h16m15.62714608s) [noise=true , address=77.90.31.13:4629 , piece_id=WON727FA2R2B5C2NEN4VMFIS76UD7TFTRAO5XJJCDYCE4MAFIVBA, seg_pos={0 0}, upload=494726]: Long tail cancellation due to optimal threshold Dec 29 03:45:15.756169287 (564h16m15.627252322s) [noise=true , address=45.91.116.14:4191 , piece_id=BMWVFCCTAKVWFA5XDGAS34QOTGQJWKVQL4J4EV4GKCFCY6DZW4DQ, seg_pos={0 0}, upload=494726]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: write tcp 188.166.225.170:37384->45.91.116.14:4191: use of closed network connection; write tcp 188.166.225.170:37384->45.91.116.14:4191: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37384->45.91.116.14:4191: use of closed network connection Dec 29 03:45:15.756185357 (564h16m15.627268402s) [noise=true , address=45.91.116.14:4191 , piece_id=BMWVFCCTAKVWFA5XDGAS34QOTGQJWKVQL4J4EV4GKCFCY6DZW4DQ, seg_pos={0 0}, upload=494726]: Long tail cancellation due to optimal threshold Dec 29 03:45:15.756342076 (564h16m15.627425111s) [noise=true , address=77.90.31.14:4662 , piece_id=ITJMRNF54I7WF6OWA4UWRZ7QXLDB6JTTB5BXVFXY3VSID3YWT5VA, seg_pos={0 0}, upload=494726]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): protocol: expected piece hash; context canceled; EOF Dec 29 03:45:15.756349179 (564h16m15.627432224s) [noise=true , address=77.90.31.14:4662 , piece_id=ITJMRNF54I7WF6OWA4UWRZ7QXLDB6JTTB5BXVFXY3VSID3YWT5VA, seg_pos={0 0}, upload=494726]: Long tail cancellation due to optimal threshold Dec 29 03:45:15.756369177 (564h16m15.627452222s) [seg_pos={0 0}, upload=494726]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 03:45:15.756459689 (564h16m15.627542724s) [upload=494726]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 03:45:15.756464498 (564h16m15.627547533s) [upload=494726]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 03:45:16.040555429 (564h16m15.911638464s) [upload=494726]: Done uploading segments... Dec 29 03:45:16.040575497 (564h16m15.911658532s) [upload=494726]: Upload finished. err: <nil> Duration: 30.003489684s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [..........................XX................X......XXX......] duration: 5.001208149s trace: https://jaeger.dev.storj.io/trace/79e92355ce966062 FAIL (2025-12-29 03:45:05 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001240129s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 29 out of 60 is failed (threshold=3) [.....XX.X.....X.X..XXXX..XXX.XX...X.X.....XXX...XXXXXXX.XX.X] duration: 30.002792877s trace: https://jaeger.dev.storj.io/trace/5f142192df72e1c5 FAIL (2025-12-29 03:44:58 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: UPTKQUPZE7F7QMX3G35WQI3NI2KA2SP6Y2TC4NVFBFPEA77QEGFQ) context deadline exceeded; piecestore: (Node ID: 1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8, Piece ID: ZDIJWI4H4OWFHIXAJ5TZZFBALJ7XVH6QQ3ZDHC3LC4C4UWP6TMHA) context deadline exceeded ======LOGS====== s... Dec 29 03:44:29.051837383 (564h15m28.922920428s) [upload=494714]: Got next segment. Inline: false Dec 29 03:44:29.051904711 (564h15m28.922987746s) [upload=494714]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 03:44:29.051911534 (564h15m28.922994569s) [upload=494714]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 03:44:29.297173569 (564h15m29.168256604s) [seg_pos={0 0}, upload=494714]: Begin upload segment... Dec 29 03:44:29.297218123 (564h15m29.168301158s) [seg_pos={0 0}, upload=494714]: Done begin upload segment. Dec 29 03:44:29.297241578 (564h15m29.168324613s) [noise=true , address=145.79.227.6:4258 , piece_id=WIAFEJ6U7W2GNQCTIBZGHHWCZXW6XK2WKLV6FEJOQMJXLDR4JL6A, seg_pos={0 0}, upload=494714]: Uploading piece... Dec 29 03:44:29.297347610 (564h15m29.168430645s) [noise=true , address=145.79.227.12:4765 , piece_id=TSGQRC24SMUY2YZEFPEGREPSJUKOF22MAMJ3P67V63Z7DLYMGU5Q, seg_pos={0 0}, upload=494714]: Uploading piece... Dec 29 03:44:29.297929508 (564h15m29.169012543s) [noise=true , address=145.79.227.45:4231 , piece_id=M6T6YMNWYTOYD34NSEQ5XQLSHTUHN5POZD3K7KERQZ4YN7STTIDA, seg_pos={0 0}, upload=494714]: Uploading piece... Dec 29 03:44:29.298568765 (564h15m29.16965181s) [noise=true , address=77.90.31.10:4520 , piece_id=3QUE3TOSMN43PILJD5UAKAG2FLB4D4LE5ID3QKXI5GQFYY3FWN5A, seg_pos={0 0}, upload=494714]: Uploading piece... Dec 29 03:44:29.298682021 (564h15m29.169765056s) [noise=true , address=45.91.116.6:4975 , piece_id=UPTKQUPZE7F7QMX3G35WQI3NI2KA2SP6Y2TC4NVFBFPEA77QEGFQ, seg_pos={0 0}, upload=494714]: Uploading piece... Dec 29 03:44:29.299278406 (564h15m29.170361441s) [noise=true , address=145.79.227.38:4978 , piece_id=MKJNYTJEKT43WDLFYDGYKSQRZG65AT6SXJMVFB63B4Z2H5R7CDSA, seg_pos={0 0}, upload=494714]: Uploading piece... Dec 29 03:44:29.299638152 (564h15m29.170721187s) [noise=true , address=45.91.116.13:4156 , piece_id=5PPZ7TEIH5OTD35K3TKTKYW5SKNUDVUVKGNBJNKIGG5UDYMZT3FA, seg_pos={0 0}, upload=494714]: Uploading piece... Dec 29 03:44:29.300050266 (564h15m29.171133301s) [noise=true , address=145.79.227.46:4601 , piece_id=B4CREHMQJK46RH7NX4FTJEKNHKI7DWTNXW4M4WOATDWMV3FWHU5A, seg_pos={0 0}, upload=494714]: Uploading piece... Dec 29 03:44:29.300517967 (564h15m29.171601002s) [noise=true , address=145.79.227.47:4265 , piece_id=XPGD7WNAI7XRCAGGET3SEDWUUXOW6VCXGJ7YIHUMPQRQFIC2VEYA, seg_pos={0 0}, upload=494714]: Uploading piece... Dec 29 03:44:29.300930653 (564h15m29.172013688s) [noise=true , address=145.79.227.48:4302 , piece_id=YJ3NUVCQF4ADI4WXBXEJRTH3BOWQHEL74RV6PW5CUBBL5HHDWI2A, seg_pos={0 0}, upload=494714]: Uploading piece... Dec 29 03:44:29.301739403 (564h15m29.172822438s) [noise=true , address=145.79.227.51:4409 , piece_id=QG4AN7HZSCCCJFCCWWMPK5OHXDAD77NQOLWMUGXSZBYET6NCQRBQ, seg_pos={0 0}, upload=494714]: Uploading piece... Dec 29 03:44:29.302226620 (564h15m29.173309655s) [noise=true , address=45.91.116.8:4047 , piece_id=F2O2Y424D6HYTIPIQ2MWZORFGIPMUGE2K5IEYWLN5HUJGZFCYQIA, seg_pos={0 0}, upload=494714]: Uploading piece... Dec 29 03:44:29.307128745 (564h15m29.17821179s) [noise=true , address=45.91.116.11:4120 , piece_id=ZDIJWI4H4OWFHIXAJ5TZZFBALJ7XVH6QQ3ZDHC3LC4C4UWP6TMHA, seg_pos={0 0}, upload=494714]: Uploading piece... Dec 29 03:44:29.308479416 (564h15m29.179562451s) [noise=true , address=145.79.227.39:4013 , piece_id=EO6XFZ6KWNEB5NJJARANK3B4OEBG2QDCZXZC7FKLAWTOBIRE2KSQ, seg_pos={0 0}, upload=494714]: Uploading piece... Dec 29 03:44:29.309718857 (564h15m29.180801892s) [noise=true , address=145.79.227.11:4730 , piece_id=F73KFHSGQYKFEL2HXG53I3UZUE7BBFD3CQAZMPNONL44A3CPTSPQ, seg_pos={0 0}, upload=494714]: Uploading piece... Dec 29 03:44:29.312494090 (564h15m29.183577126s) [noise=true , address=77.90.31.16:4736 , piece_id=B7G4T37IU4RPBRHANOSPUIV6XD3DDKC47HLCJBJXZ5NS2S5VMG3Q, seg_pos={0 0}, upload=494714]: Uploading piece... Dec 29 03:44:30.993967266 (564h15m30.865050302s) [upload=494714]: Next returned nil segment Dec 29 03:44:30.993988026 (564h15m30.865071061s) [upload=494714]: Waiting for error group managing segments... Dec 29 03:44:31.274004057 (564h15m31.145087102s) [noise=true , address=145.79.227.6:4258 , piece_id=WIAFEJ6U7W2GNQCTIBZGHHWCZXW6XK2WKLV6FEJOQMJXLDR4JL6A, seg_pos={0 0}, upload=494714]: Done uploading piece. err: <nil> Dec 29 03:44:31.291922747 (564h15m31.163005792s) [noise=true , address=145.79.227.48:4302 , piece_id=YJ3NUVCQF4ADI4WXBXEJRTH3BOWQHEL74RV6PW5CUBBL5HHDWI2A, seg_pos={0 0}, upload=494714]: Done uploading piece. err: <nil> Dec 29 03:44:31.298011270 (564h15m31.169094305s) [noise=true , address=145.79.227.47:4265 , piece_id=XPGD7WNAI7XRCAGGET3SEDWUUXOW6VCXGJ7YIHUMPQRQFIC2VEYA, seg_pos={0 0}, upload=494714]: Done uploading piece. err: <nil> Dec 29 03:44:31.298441890 (564h15m31.169524935s) [noise=true , address=145.79.227.46:4601 , piece_id=B4CREHMQJK46RH7NX4FTJEKNHKI7DWTNXW4M4WOATDWMV3FWHU5A, seg_pos={0 0}, upload=494714]: Done uploading piece. err: <nil> Dec 29 03:44:31.298907336 (564h15m31.169990371s) [noise=true , address=145.79.227.38:4978 , piece_id=MKJNYTJEKT43WDLFYDGYKSQRZG65AT6SXJMVFB63B4Z2H5R7CDSA, seg_pos={0 0}, upload=494714]: Done uploading piece. err: <nil> Dec 29 03:44:31.301493530 (564h15m31.172576575s) [noise=true , address=145.79.227.12:4765 , piece_id=TSGQRC24SMUY2YZEFPEGREPSJUKOF22MAMJ3P67V63Z7DLYMGU5Q, seg_pos={0 0}, upload=494714]: Done uploading piece. err: <nil> Dec 29 03:44:31.323392540 (564h15m31.194475585s) [noise=true , address=145.79.227.45:4231 , piece_id=M6T6YMNWYTOYD34NSEQ5XQLSHTUHN5POZD3K7KERQZ4YN7STTIDA, seg_pos={0 0}, upload=494714]: Done uploading piece. err: <nil> Dec 29 03:44:31.325383140 (564h15m31.196466185s) [noise=true , address=145.79.227.39:4013 , piece_id=EO6XFZ6KWNEB5NJJARANK3B4OEBG2QDCZXZC7FKLAWTOBIRE2KSQ, seg_pos={0 0}, upload=494714]: Done uploading piece. err: <nil> Dec 29 03:44:31.326066651 (564h15m31.197149686s) [noise=true , address=145.79.227.11:4730 , piece_id=F73KFHSGQYKFEL2HXG53I3UZUE7BBFD3CQAZMPNONL44A3CPTSPQ, seg_pos={0 0}, upload=494714]: Done uploading piece. err: <nil> Dec 29 03:44:31.583415550 (564h15m31.454498585s) [noise=true , address=45.91.116.6:4975 , piece_id=UPTKQUPZE7F7QMX3G35WQI3NI2KA2SP6Y2TC4NVFBFPEA77QEGFQ, seg_pos={0 0}, upload=494714]: Done uploading piece. err: <nil> Dec 29 03:44:31.587186019 (564h15m31.458269054s) [noise=true , address=45.91.116.11:4120 , piece_id=ZDIJWI4H4OWFHIXAJ5TZZFBALJ7XVH6QQ3ZDHC3LC4C4UWP6TMHA, seg_pos={0 0}, upload=494714]: Done uploading piece. err: <nil> Dec 29 03:44:31.647916038 (564h15m31.518999073s) [noise=true , address=45.91.116.13:4156 , piece_id=5PPZ7TEIH5OTD35K3TKTKYW5SKNUDVUVKGNBJNKIGG5UDYMZT3FA, seg_pos={0 0}, upload=494714]: Done uploading piece. err: <nil> Dec 29 03:44:31.989739910 (564h15m31.860822945s) [noise=true , address=77.90.31.16:4736 , piece_id=B7G4T37IU4RPBRHANOSPUIV6XD3DDKC47HLCJBJXZ5NS2S5VMG3Q, seg_pos={0 0}, upload=494714]: Done uploading piece. err: <nil> Dec 29 03:44:31.989763615 (564h15m31.86084665s) [seg_pos={0 0}, upload=494714]: Segment reached optimal threshold of 13 pieces. Dec 29 03:44:31.990063807 (564h15m31.861146852s) [noise=true , address=45.91.116.8:4047 , piece_id=F2O2Y424D6HYTIPIQ2MWZORFGIPMUGE2K5IEYWLN5HUJGZFCYQIA, seg_pos={0 0}, upload=494714]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: write tcp 188.166.225.170:42540->45.91.116.8:4047: use of closed network connection; write tcp 188.166.225.170:42540->45.91.116.8:4047: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42540->45.91.116.8:4047: use of closed network connection Dec 29 03:44:31.990075078 (564h15m31.861158123s) [noise=true , address=45.91.116.8:4047 , piece_id=F2O2Y424D6HYTIPIQ2MWZORFGIPMUGE2K5IEYWLN5HUJGZFCYQIA, seg_pos={0 0}, upload=494714]: Long tail cancellation due to optimal threshold Dec 29 03:44:31.990212480 (564h15m31.861295515s) [noise=true , address=77.90.31.10:4520 , piece_id=3QUE3TOSMN43PILJD5UAKAG2FLB4D4LE5ID3QKXI5GQFYY3FWN5A, seg_pos={0 0}, upload=494714]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp): protocol: expected piece hash; context canceled; EOF Dec 29 03:44:31.990221136 (564h15m31.861304191s) [noise=true , address=77.90.31.10:4520 , piece_id=3QUE3TOSMN43PILJD5UAKAG2FLB4D4LE5ID3QKXI5GQFYY3FWN5A, seg_pos={0 0}, upload=494714]: Long tail cancellation due to optimal threshold Dec 29 03:44:31.990297412 (564h15m31.861380447s) [noise=true , address=145.79.227.51:4409 , piece_id=QG4AN7HZSCCCJFCCWWMPK5OHXDAD77NQOLWMUGXSZBYET6NCQRBQ, seg_pos={0 0}, upload=494714]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UoCGA4FjE4yADPXL1oifSPJssPgSGh81REZHwGv2eu1Y1kEaj): protocol: write tcp 188.166.225.170:60732->145.79.227.51:4409: use of closed network connection; write tcp 188.166.225.170:60732->145.79.227.51:4409: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60732->145.79.227.51:4409: use of closed network connection Dec 29 03:44:31.990305507 (564h15m31.861388552s) [noise=true , address=145.79.227.51:4409 , piece_id=QG4AN7HZSCCCJFCCWWMPK5OHXDAD77NQOLWMUGXSZBYET6NCQRBQ, seg_pos={0 0}, upload=494714]: Long tail cancellation due to optimal threshold Dec 29 03:44:31.990325174 (564h15m31.861408209s) [seg_pos={0 0}, upload=494714]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 03:44:31.990400898 (564h15m31.861483943s) [upload=494714]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 03:44:31.990408724 (564h15m31.861491789s) [upload=494714]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 03:44:32.283757000 (564h15m32.154840045s) [upload=494714]: Done uploading segments... Dec 29 03:44:32.283780526 (564h15m32.154863561s) [upload=494714]: Upload finished. err: <nil> Duration: 30.002810372s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [................................X...................X.......] duration: 41.243892581s trace: https://jaeger.dev.storj.io/trace/2408171d336afafd FAIL (2025-12-29 03:44:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12Tz3HEHACmHU9Gp2W2vHVuF5PXNmdsa5N8nYhwt52HCTjggqRD, Piece ID: S3MEER2QCVCORYQNYNY2S22TC2H5HHKZW7VQJ54VL754UZTD42JQ) context deadline exceeded; piecestore: (Node ID: 1ve7KKYDbNo8Ma7wGVBH5x9z7zbPtPAPztmuQ9r6ijPpVgh5NX, Piece ID: S7T55BAWP53YJVHPW2GPLUM2VB3NBHZ2YD4UWBKKWCFHDAXAW3KQ) context deadline exceeded; piecestore: (Node ID: 1WVHPacX35JJ8PFzCs3UtGwsR4yVHXKmiqfGedK9KfXbHNJtmu, Piece ID: NYWKOTPLBZQIHGSQFMTT2QE6OQX7H5OKFVWJK3I65CWGGDSDDBMA) context deadline exceeded ======LOGS====== upload=494622]: Uploading piece... Dec 29 03:37:43.315384541 (564h8m43.186467596s) [noise=true , address=145.79.227.11:4762 , piece_id=2GQ5ASSBMUC52NCYRLBF3EOKYEA3G6DTIA6K56DE3OUQ4F5OLX2Q, seg_pos={0 0}, upload=494622]: Uploading piece... Dec 29 03:37:43.315899872 (564h8m43.186982907s) [noise=true , address=77.90.31.13:4649 , piece_id=S7T55BAWP53YJVHPW2GPLUM2VB3NBHZ2YD4UWBKKWCFHDAXAW3KQ, seg_pos={0 0}, upload=494622]: Uploading piece... Dec 29 03:37:43.316395165 (564h8m43.1874782s) [noise=true , address=77.90.31.18:4834 , piece_id=FQS6BFFJTORJPKJCINDZKXTY6GFRQTR3FZPWAEG7F2DNROUEPGJA, seg_pos={0 0}, upload=494622]: Uploading piece... Dec 29 03:37:43.316885961 (564h8m43.187969006s) [noise=true , address=193.149.164.143:4695 , piece_id=YF4EWRMS3GWEYPSA67GLG6EA5H7GZ55GTA3ISIEJ7RFEG2NDTLEQ, seg_pos={0 0}, upload=494622]: Uploading piece... Dec 29 03:37:43.317355795 (564h8m43.18843883s) [noise=true , address=145.79.227.7:4297 , piece_id=NNJ2KDDLNZHT4SGOUXBWBYD5OGW6MTGCAWZ63D5LH7BGM6KOJGDA, seg_pos={0 0}, upload=494622]: Uploading piece... Dec 29 03:37:43.317364631 (564h8m43.188447666s) [noise=true , address=145.79.227.33:4886 , piece_id=DB64UXSDC47SRTQX7PJGE66RCL7XFW3TGMCU3JOWVEJVGVHLYGWQ, seg_pos={0 0}, upload=494622]: Uploading piece... Dec 29 03:37:43.317689451 (564h8m43.188772486s) [noise=true , address=64.49.13.26:4950 , piece_id=HQ6BBIFQAD5RRN7FRWGDFHAVVMSDUWJUGTX7LBTA2TQJ3PNWKUHA, seg_pos={0 0}, upload=494622]: Uploading piece... Dec 29 03:37:43.317877448 (564h8m43.188960493s) [noise=true , address=45.91.116.10:4096 , piece_id=VIQK6EV67DONFZY77BOTWHY7POJEOE7FDHB6TKQMHVSHJJM35V7Q, seg_pos={0 0}, upload=494622]: Uploading piece... Dec 29 03:37:43.318315813 (564h8m43.189398858s) [noise=true , address=145.79.227.9:4388 , piece_id=DQU5TSAWTEEIBT272I76ZMGXD6FJT2QMXQISRDMWG5NOPORLBLLA, seg_pos={0 0}, upload=494622]: Uploading piece... Dec 29 03:37:43.318722317 (564h8m43.189805352s) [noise=true , address=77.90.31.12:4600 , piece_id=N4KXD5SYPYOKBXQRSL3UKNDDV7XB4KOKRYRYKGMMAHMAO6VVOEHQ, seg_pos={0 0}, upload=494622]: Uploading piece... Dec 29 03:37:43.319195248 (564h8m43.190278293s) [noise=true , address=45.91.116.8:4050 , piece_id=NYWKOTPLBZQIHGSQFMTT2QE6OQX7H5OKFVWJK3I65CWGGDSDDBMA, seg_pos={0 0}, upload=494622]: Uploading piece... Dec 29 03:37:43.319666174 (564h8m43.190749219s) [noise=true , address=77.90.31.21:4899 , piece_id=4MZKL4KVJF4DYGVPYGVOUHD6TYIDCRER33YPW6RG7KT6B6IXQEOA, seg_pos={0 0}, upload=494622]: Uploading piece... Dec 29 03:37:43.321557184 (564h8m43.192640229s) [noise=true , address=145.79.227.34:4909 , piece_id=UCGJPN357USFHIQ7II5GLRKNDE4H4HUVDQ7CB5R27C3VGRP2R3IA, seg_pos={0 0}, upload=494622]: Uploading piece... Dec 29 03:37:43.322231088 (564h8m43.193314133s) [noise=true , address=45.91.116.11:4139 , piece_id=S3MEER2QCVCORYQNYNY2S22TC2H5HHKZW7VQJ54VL754UZTD42JQ, seg_pos={0 0}, upload=494622]: Uploading piece... Dec 29 03:37:43.322851720 (564h8m43.193934755s) [noise=true , address=145.79.227.51:4444 , piece_id=EWEXXYYWF43J7FG6NPSJ2PICHTRPY533L4DQQX2CSW5IMJ6CB2PQ, seg_pos={0 0}, upload=494622]: Uploading piece... Dec 29 03:37:43.323435972 (564h8m43.194519007s) [noise=true , address=145.79.227.40:4068 , piece_id=4QOJWRVEE3ERM2AHP764XSTE7ZYHUUJ2ENBVXRS5UDIGKZOK7YKA, seg_pos={0 0}, upload=494622]: Uploading piece... Dec 29 03:37:43.323901118 (564h8m43.194984153s) [noise=true , address=145.79.227.30:4790 , piece_id=2TSIHAVWOZPOVKSTG2ZXBBEFEMMMRLMOWHM47TDANARFPDJM3NPQ, seg_pos={0 0}, upload=494622]: Uploading piece... Dec 29 03:37:44.620573279 (564h8m44.491656314s) [upload=494622]: Next returned nil segment Dec 29 03:37:44.620593617 (564h8m44.491676652s) [upload=494622]: Waiting for error group managing segments... Dec 29 03:37:45.178264102 (564h8m45.049347137s) [noise=true , address=145.79.227.11:4762 , piece_id=2GQ5ASSBMUC52NCYRLBF3EOKYEA3G6DTIA6K56DE3OUQ4F5OLX2Q, seg_pos={0 0}, upload=494622]: Done uploading piece. err: <nil> Dec 29 03:37:45.190189951 (564h8m45.061272986s) [noise=true , address=145.79.227.51:4444 , piece_id=EWEXXYYWF43J7FG6NPSJ2PICHTRPY533L4DQQX2CSW5IMJ6CB2PQ, seg_pos={0 0}, upload=494622]: Done uploading piece. err: <nil> Dec 29 03:37:45.199163818 (564h8m45.070246853s) [noise=true , address=145.79.227.7:4297 , piece_id=NNJ2KDDLNZHT4SGOUXBWBYD5OGW6MTGCAWZ63D5LH7BGM6KOJGDA, seg_pos={0 0}, upload=494622]: Done uploading piece. err: <nil> Dec 29 03:37:45.448727893 (564h8m45.319810928s) [noise=true , address=45.91.116.11:4139 , piece_id=S3MEER2QCVCORYQNYNY2S22TC2H5HHKZW7VQJ54VL754UZTD42JQ, seg_pos={0 0}, upload=494622]: Done uploading piece. err: <nil> Dec 29 03:37:45.459119140 (564h8m45.330202175s) [noise=true , address=77.90.31.13:4649 , piece_id=S7T55BAWP53YJVHPW2GPLUM2VB3NBHZ2YD4UWBKKWCFHDAXAW3KQ, seg_pos={0 0}, upload=494622]: Done uploading piece. err: <nil> Dec 29 03:37:45.471609714 (564h8m45.342692759s) [noise=true , address=45.91.116.8:4050 , piece_id=NYWKOTPLBZQIHGSQFMTT2QE6OQX7H5OKFVWJK3I65CWGGDSDDBMA, seg_pos={0 0}, upload=494622]: Done uploading piece. err: <nil> Dec 29 03:37:45.489098314 (564h8m45.36018136s) [noise=true , address=77.90.31.21:4899 , piece_id=4MZKL4KVJF4DYGVPYGVOUHD6TYIDCRER33YPW6RG7KT6B6IXQEOA, seg_pos={0 0}, upload=494622]: Done uploading piece. err: <nil> Dec 29 03:37:45.489726060 (564h8m45.360809105s) [noise=true , address=77.90.31.11:4574 , piece_id=3TP2ME5NTXVK45B34W5U4DESLFJLOJEPRQOJLEEPN2PIMYS2IDWQ, seg_pos={0 0}, upload=494622]: Done uploading piece. err: <nil> Dec 29 03:37:45.779669865 (564h8m45.65075291s) [noise=true , address=145.79.227.9:4388 , piece_id=DQU5TSAWTEEIBT272I76ZMGXD6FJT2QMXQISRDMWG5NOPORLBLLA, seg_pos={0 0}, upload=494622]: Done uploading piece. err: <nil> Dec 29 03:37:45.817564973 (564h8m45.688648008s) [noise=true , address=77.90.31.18:4834 , piece_id=FQS6BFFJTORJPKJCINDZKXTY6GFRQTR3FZPWAEG7F2DNROUEPGJA, seg_pos={0 0}, upload=494622]: Done uploading piece. err: <nil> Dec 29 03:37:46.001090802 (564h8m45.872173857s) [noise=true , address=193.149.164.143:4695 , piece_id=YF4EWRMS3GWEYPSA67GLG6EA5H7GZ55GTA3ISIEJ7RFEG2NDTLEQ, seg_pos={0 0}, upload=494622]: Done uploading piece. err: <nil> Dec 29 03:37:46.129753764 (564h8m46.000836819s) [noise=true , address=145.79.227.33:4886 , piece_id=DB64UXSDC47SRTQX7PJGE66RCL7XFW3TGMCU3JOWVEJVGVHLYGWQ, seg_pos={0 0}, upload=494622]: Done uploading piece. err: <nil> Dec 29 03:37:46.163280064 (564h8m46.034363109s) [noise=true , address=145.79.227.34:4909 , piece_id=UCGJPN357USFHIQ7II5GLRKNDE4H4HUVDQ7CB5R27C3VGRP2R3IA, seg_pos={0 0}, upload=494622]: Done uploading piece. err: <nil> Dec 29 03:37:46.167742170 (564h8m46.038825205s) [noise=true , address=145.79.227.30:4790 , piece_id=2TSIHAVWOZPOVKSTG2ZXBBEFEMMMRLMOWHM47TDANARFPDJM3NPQ, seg_pos={0 0}, upload=494622]: Done uploading piece. err: <nil> Dec 29 03:37:46.167764403 (564h8m46.038847448s) [seg_pos={0 0}, upload=494622]: Segment reached optimal threshold of 14 pieces. Dec 29 03:37:46.167996534 (564h8m46.039079569s) [noise=true , address=45.91.116.10:4096 , piece_id=VIQK6EV67DONFZY77BOTWHY7POJEOE7FDHB6TKQMHVSHJJM35V7Q, seg_pos={0 0}, upload=494622]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12MZX1GwZGrEi2q44dPiYrwMoum5A2caZtXTS3qiLNcoyHhon7p): protocol: expected piece hash; context canceled; EOF Dec 29 03:37:46.168005812 (564h8m46.039088857s) [noise=true , address=45.91.116.10:4096 , piece_id=VIQK6EV67DONFZY77BOTWHY7POJEOE7FDHB6TKQMHVSHJJM35V7Q, seg_pos={0 0}, upload=494622]: Long tail cancellation due to optimal threshold Dec 29 03:37:46.168120160 (564h8m46.039203195s) [noise=true , address=64.49.13.26:4950 , piece_id=HQ6BBIFQAD5RRN7FRWGDFHAVVMSDUWJUGTX7LBTA2TQJ3PNWKUHA, seg_pos={0 0}, upload=494622]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14Tv25fLSqTnZQiDzrYqKCSbdt6ep3upgqD2Se8QSRuQFynkSN): protocol: expected piece hash; context canceled; EOF Dec 29 03:37:46.168136090 (564h8m46.039219125s) [noise=true , address=64.49.13.26:4950 , piece_id=HQ6BBIFQAD5RRN7FRWGDFHAVVMSDUWJUGTX7LBTA2TQJ3PNWKUHA, seg_pos={0 0}, upload=494622]: Long tail cancellation due to optimal threshold Dec 29 03:37:46.168274995 (564h8m46.03935804s) [noise=true , address=77.90.31.12:4600 , piece_id=N4KXD5SYPYOKBXQRSL3UKNDDV7XB4KOKRYRYKGMMAHMAO6VVOEHQ, seg_pos={0 0}, upload=494622]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PKT2EcU6jAe6oUUdiTcwxGQsFa36uEmown77SemqQMM5Xw5nc): protocol: expected piece hash; context canceled; EOF Dec 29 03:37:46.168283491 (564h8m46.039366536s) [noise=true , address=77.90.31.12:4600 , piece_id=N4KXD5SYPYOKBXQRSL3UKNDDV7XB4KOKRYRYKGMMAHMAO6VVOEHQ, seg_pos={0 0}, upload=494622]: Long tail cancellation due to optimal threshold Dec 29 03:37:46.168394032 (564h8m46.039477067s) [noise=true , address=145.79.227.40:4068 , piece_id=4QOJWRVEE3ERM2AHP764XSTE7ZYHUUJ2ENBVXRS5UDIGKZOK7YKA, seg_pos={0 0}, upload=494622]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Lh9dYpHCxbs7gpZ4Xt7os5Gm5Nco6jrNW1V1FHddVUEMkFfPY): protocol: expected piece hash; context canceled; EOF Dec 29 03:37:46.168403810 (564h8m46.039486845s) [noise=true , address=145.79.227.40:4068 , piece_id=4QOJWRVEE3ERM2AHP764XSTE7ZYHUUJ2ENBVXRS5UDIGKZOK7YKA, seg_pos={0 0}, upload=494622]: Long tail cancellation due to optimal threshold Dec 29 03:37:46.168432725 (564h8m46.03951577s) [seg_pos={0 0}, upload=494622]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 03:37:46.168576960 (564h8m46.039659995s) [upload=494622]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 03:37:46.168584033 (564h8m46.039667068s) [upload=494622]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 03:37:46.466683658 (564h8m46.337766723s) [upload=494622]: Done uploading segments... Dec 29 03:37:46.466712052 (564h8m46.337795087s) [upload=494622]: Upload finished. err: <nil> Duration: 41.243919132s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.................................X...................X......] duration: 48.22843962s trace: https://jaeger.dev.storj.io/trace/2408171d336afafd FAIL (2025-12-29 03:44:10 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12Tz3HEHACmHU9Gp2W2vHVuF5PXNmdsa5N8nYhwt52HCTjggqRD, Piece ID: S3MEER2QCVCORYQNYNY2S22TC2H5HHKZW7VQJ54VL754UZTD42JQ) context deadline exceeded; piecestore: (Node ID: 1ve7KKYDbNo8Ma7wGVBH5x9z7zbPtPAPztmuQ9r6ijPpVgh5NX, Piece ID: S7T55BAWP53YJVHPW2GPLUM2VB3NBHZ2YD4UWBKKWCFHDAXAW3KQ) context deadline exceeded; piecestore: (Node ID: 1WVHPacX35JJ8PFzCs3UtGwsR4yVHXKmiqfGedK9KfXbHNJtmu, Piece ID: NYWKOTPLBZQIHGSQFMTT2QE6OQX7H5OKFVWJK3I65CWGGDSDDBMA) context deadline exceeded ======LOGS====== upload=494622]: Uploading piece... Dec 29 03:37:43.315384541 (564h8m43.186467596s) [noise=true , address=145.79.227.11:4762 , piece_id=2GQ5ASSBMUC52NCYRLBF3EOKYEA3G6DTIA6K56DE3OUQ4F5OLX2Q, seg_pos={0 0}, upload=494622]: Uploading piece... Dec 29 03:37:43.315899872 (564h8m43.186982907s) [noise=true , address=77.90.31.13:4649 , piece_id=S7T55BAWP53YJVHPW2GPLUM2VB3NBHZ2YD4UWBKKWCFHDAXAW3KQ, seg_pos={0 0}, upload=494622]: Uploading piece... Dec 29 03:37:43.316395165 (564h8m43.1874782s) [noise=true , address=77.90.31.18:4834 , piece_id=FQS6BFFJTORJPKJCINDZKXTY6GFRQTR3FZPWAEG7F2DNROUEPGJA, seg_pos={0 0}, upload=494622]: Uploading piece... Dec 29 03:37:43.316885961 (564h8m43.187969006s) [noise=true , address=193.149.164.143:4695 , piece_id=YF4EWRMS3GWEYPSA67GLG6EA5H7GZ55GTA3ISIEJ7RFEG2NDTLEQ, seg_pos={0 0}, upload=494622]: Uploading piece... Dec 29 03:37:43.317355795 (564h8m43.18843883s) [noise=true , address=145.79.227.7:4297 , piece_id=NNJ2KDDLNZHT4SGOUXBWBYD5OGW6MTGCAWZ63D5LH7BGM6KOJGDA, seg_pos={0 0}, upload=494622]: Uploading piece... Dec 29 03:37:43.317364631 (564h8m43.188447666s) [noise=true , address=145.79.227.33:4886 , piece_id=DB64UXSDC47SRTQX7PJGE66RCL7XFW3TGMCU3JOWVEJVGVHLYGWQ, seg_pos={0 0}, upload=494622]: Uploading piece... Dec 29 03:37:43.317689451 (564h8m43.188772486s) [noise=true , address=64.49.13.26:4950 , piece_id=HQ6BBIFQAD5RRN7FRWGDFHAVVMSDUWJUGTX7LBTA2TQJ3PNWKUHA, seg_pos={0 0}, upload=494622]: Uploading piece... Dec 29 03:37:43.317877448 (564h8m43.188960493s) [noise=true , address=45.91.116.10:4096 , piece_id=VIQK6EV67DONFZY77BOTWHY7POJEOE7FDHB6TKQMHVSHJJM35V7Q, seg_pos={0 0}, upload=494622]: Uploading piece... Dec 29 03:37:43.318315813 (564h8m43.189398858s) [noise=true , address=145.79.227.9:4388 , piece_id=DQU5TSAWTEEIBT272I76ZMGXD6FJT2QMXQISRDMWG5NOPORLBLLA, seg_pos={0 0}, upload=494622]: Uploading piece... Dec 29 03:37:43.318722317 (564h8m43.189805352s) [noise=true , address=77.90.31.12:4600 , piece_id=N4KXD5SYPYOKBXQRSL3UKNDDV7XB4KOKRYRYKGMMAHMAO6VVOEHQ, seg_pos={0 0}, upload=494622]: Uploading piece... Dec 29 03:37:43.319195248 (564h8m43.190278293s) [noise=true , address=45.91.116.8:4050 , piece_id=NYWKOTPLBZQIHGSQFMTT2QE6OQX7H5OKFVWJK3I65CWGGDSDDBMA, seg_pos={0 0}, upload=494622]: Uploading piece... Dec 29 03:37:43.319666174 (564h8m43.190749219s) [noise=true , address=77.90.31.21:4899 , piece_id=4MZKL4KVJF4DYGVPYGVOUHD6TYIDCRER33YPW6RG7KT6B6IXQEOA, seg_pos={0 0}, upload=494622]: Uploading piece... Dec 29 03:37:43.321557184 (564h8m43.192640229s) [noise=true , address=145.79.227.34:4909 , piece_id=UCGJPN357USFHIQ7II5GLRKNDE4H4HUVDQ7CB5R27C3VGRP2R3IA, seg_pos={0 0}, upload=494622]: Uploading piece... Dec 29 03:37:43.322231088 (564h8m43.193314133s) [noise=true , address=45.91.116.11:4139 , piece_id=S3MEER2QCVCORYQNYNY2S22TC2H5HHKZW7VQJ54VL754UZTD42JQ, seg_pos={0 0}, upload=494622]: Uploading piece... Dec 29 03:37:43.322851720 (564h8m43.193934755s) [noise=true , address=145.79.227.51:4444 , piece_id=EWEXXYYWF43J7FG6NPSJ2PICHTRPY533L4DQQX2CSW5IMJ6CB2PQ, seg_pos={0 0}, upload=494622]: Uploading piece... Dec 29 03:37:43.323435972 (564h8m43.194519007s) [noise=true , address=145.79.227.40:4068 , piece_id=4QOJWRVEE3ERM2AHP764XSTE7ZYHUUJ2ENBVXRS5UDIGKZOK7YKA, seg_pos={0 0}, upload=494622]: Uploading piece... Dec 29 03:37:43.323901118 (564h8m43.194984153s) [noise=true , address=145.79.227.30:4790 , piece_id=2TSIHAVWOZPOVKSTG2ZXBBEFEMMMRLMOWHM47TDANARFPDJM3NPQ, seg_pos={0 0}, upload=494622]: Uploading piece... Dec 29 03:37:44.620573279 (564h8m44.491656314s) [upload=494622]: Next returned nil segment Dec 29 03:37:44.620593617 (564h8m44.491676652s) [upload=494622]: Waiting for error group managing segments... Dec 29 03:37:45.178264102 (564h8m45.049347137s) [noise=true , address=145.79.227.11:4762 , piece_id=2GQ5ASSBMUC52NCYRLBF3EOKYEA3G6DTIA6K56DE3OUQ4F5OLX2Q, seg_pos={0 0}, upload=494622]: Done uploading piece. err: <nil> Dec 29 03:37:45.190189951 (564h8m45.061272986s) [noise=true , address=145.79.227.51:4444 , piece_id=EWEXXYYWF43J7FG6NPSJ2PICHTRPY533L4DQQX2CSW5IMJ6CB2PQ, seg_pos={0 0}, upload=494622]: Done uploading piece. err: <nil> Dec 29 03:37:45.199163818 (564h8m45.070246853s) [noise=true , address=145.79.227.7:4297 , piece_id=NNJ2KDDLNZHT4SGOUXBWBYD5OGW6MTGCAWZ63D5LH7BGM6KOJGDA, seg_pos={0 0}, upload=494622]: Done uploading piece. err: <nil> Dec 29 03:37:45.448727893 (564h8m45.319810928s) [noise=true , address=45.91.116.11:4139 , piece_id=S3MEER2QCVCORYQNYNY2S22TC2H5HHKZW7VQJ54VL754UZTD42JQ, seg_pos={0 0}, upload=494622]: Done uploading piece. err: <nil> Dec 29 03:37:45.459119140 (564h8m45.330202175s) [noise=true , address=77.90.31.13:4649 , piece_id=S7T55BAWP53YJVHPW2GPLUM2VB3NBHZ2YD4UWBKKWCFHDAXAW3KQ, seg_pos={0 0}, upload=494622]: Done uploading piece. err: <nil> Dec 29 03:37:45.471609714 (564h8m45.342692759s) [noise=true , address=45.91.116.8:4050 , piece_id=NYWKOTPLBZQIHGSQFMTT2QE6OQX7H5OKFVWJK3I65CWGGDSDDBMA, seg_pos={0 0}, upload=494622]: Done uploading piece. err: <nil> Dec 29 03:37:45.489098314 (564h8m45.36018136s) [noise=true , address=77.90.31.21:4899 , piece_id=4MZKL4KVJF4DYGVPYGVOUHD6TYIDCRER33YPW6RG7KT6B6IXQEOA, seg_pos={0 0}, upload=494622]: Done uploading piece. err: <nil> Dec 29 03:37:45.489726060 (564h8m45.360809105s) [noise=true , address=77.90.31.11:4574 , piece_id=3TP2ME5NTXVK45B34W5U4DESLFJLOJEPRQOJLEEPN2PIMYS2IDWQ, seg_pos={0 0}, upload=494622]: Done uploading piece. err: <nil> Dec 29 03:37:45.779669865 (564h8m45.65075291s) [noise=true , address=145.79.227.9:4388 , piece_id=DQU5TSAWTEEIBT272I76ZMGXD6FJT2QMXQISRDMWG5NOPORLBLLA, seg_pos={0 0}, upload=494622]: Done uploading piece. err: <nil> Dec 29 03:37:45.817564973 (564h8m45.688648008s) [noise=true , address=77.90.31.18:4834 , piece_id=FQS6BFFJTORJPKJCINDZKXTY6GFRQTR3FZPWAEG7F2DNROUEPGJA, seg_pos={0 0}, upload=494622]: Done uploading piece. err: <nil> Dec 29 03:37:46.001090802 (564h8m45.872173857s) [noise=true , address=193.149.164.143:4695 , piece_id=YF4EWRMS3GWEYPSA67GLG6EA5H7GZ55GTA3ISIEJ7RFEG2NDTLEQ, seg_pos={0 0}, upload=494622]: Done uploading piece. err: <nil> Dec 29 03:37:46.129753764 (564h8m46.000836819s) [noise=true , address=145.79.227.33:4886 , piece_id=DB64UXSDC47SRTQX7PJGE66RCL7XFW3TGMCU3JOWVEJVGVHLYGWQ, seg_pos={0 0}, upload=494622]: Done uploading piece. err: <nil> Dec 29 03:37:46.163280064 (564h8m46.034363109s) [noise=true , address=145.79.227.34:4909 , piece_id=UCGJPN357USFHIQ7II5GLRKNDE4H4HUVDQ7CB5R27C3VGRP2R3IA, seg_pos={0 0}, upload=494622]: Done uploading piece. err: <nil> Dec 29 03:37:46.167742170 (564h8m46.038825205s) [noise=true , address=145.79.227.30:4790 , piece_id=2TSIHAVWOZPOVKSTG2ZXBBEFEMMMRLMOWHM47TDANARFPDJM3NPQ, seg_pos={0 0}, upload=494622]: Done uploading piece. err: <nil> Dec 29 03:37:46.167764403 (564h8m46.038847448s) [seg_pos={0 0}, upload=494622]: Segment reached optimal threshold of 14 pieces. Dec 29 03:37:46.167996534 (564h8m46.039079569s) [noise=true , address=45.91.116.10:4096 , piece_id=VIQK6EV67DONFZY77BOTWHY7POJEOE7FDHB6TKQMHVSHJJM35V7Q, seg_pos={0 0}, upload=494622]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12MZX1GwZGrEi2q44dPiYrwMoum5A2caZtXTS3qiLNcoyHhon7p): protocol: expected piece hash; context canceled; EOF Dec 29 03:37:46.168005812 (564h8m46.039088857s) [noise=true , address=45.91.116.10:4096 , piece_id=VIQK6EV67DONFZY77BOTWHY7POJEOE7FDHB6TKQMHVSHJJM35V7Q, seg_pos={0 0}, upload=494622]: Long tail cancellation due to optimal threshold Dec 29 03:37:46.168120160 (564h8m46.039203195s) [noise=true , address=64.49.13.26:4950 , piece_id=HQ6BBIFQAD5RRN7FRWGDFHAVVMSDUWJUGTX7LBTA2TQJ3PNWKUHA, seg_pos={0 0}, upload=494622]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14Tv25fLSqTnZQiDzrYqKCSbdt6ep3upgqD2Se8QSRuQFynkSN): protocol: expected piece hash; context canceled; EOF Dec 29 03:37:46.168136090 (564h8m46.039219125s) [noise=true , address=64.49.13.26:4950 , piece_id=HQ6BBIFQAD5RRN7FRWGDFHAVVMSDUWJUGTX7LBTA2TQJ3PNWKUHA, seg_pos={0 0}, upload=494622]: Long tail cancellation due to optimal threshold Dec 29 03:37:46.168274995 (564h8m46.03935804s) [noise=true , address=77.90.31.12:4600 , piece_id=N4KXD5SYPYOKBXQRSL3UKNDDV7XB4KOKRYRYKGMMAHMAO6VVOEHQ, seg_pos={0 0}, upload=494622]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PKT2EcU6jAe6oUUdiTcwxGQsFa36uEmown77SemqQMM5Xw5nc): protocol: expected piece hash; context canceled; EOF Dec 29 03:37:46.168283491 (564h8m46.039366536s) [noise=true , address=77.90.31.12:4600 , piece_id=N4KXD5SYPYOKBXQRSL3UKNDDV7XB4KOKRYRYKGMMAHMAO6VVOEHQ, seg_pos={0 0}, upload=494622]: Long tail cancellation due to optimal threshold Dec 29 03:37:46.168394032 (564h8m46.039477067s) [noise=true , address=145.79.227.40:4068 , piece_id=4QOJWRVEE3ERM2AHP764XSTE7ZYHUUJ2ENBVXRS5UDIGKZOK7YKA, seg_pos={0 0}, upload=494622]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Lh9dYpHCxbs7gpZ4Xt7os5Gm5Nco6jrNW1V1FHddVUEMkFfPY): protocol: expected piece hash; context canceled; EOF Dec 29 03:37:46.168403810 (564h8m46.039486845s) [noise=true , address=145.79.227.40:4068 , piece_id=4QOJWRVEE3ERM2AHP764XSTE7ZYHUUJ2ENBVXRS5UDIGKZOK7YKA, seg_pos={0 0}, upload=494622]: Long tail cancellation due to optimal threshold Dec 29 03:37:46.168432725 (564h8m46.03951577s) [seg_pos={0 0}, upload=494622]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 03:37:46.168576960 (564h8m46.039659995s) [upload=494622]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 03:37:46.168584033 (564h8m46.039667068s) [upload=494622]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 03:37:46.466683658 (564h8m46.337766723s) [upload=494622]: Done uploading segments... Dec 29 03:37:46.466712052 (564h8m46.337795087s) [upload=494622]: Upload finished. err: <nil> Duration: 48.228458976s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [...........................XX................X......XXX.....] duration: 3.774342003s trace: https://jaeger.dev.storj.io/trace/79e92355ce966062 FAIL (2025-12-29 03:43:51 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.774369795s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 29 out of 60 is failed (threshold=3) [X.....XX.X.....X.X..XXXX..XXX.XX...X.X.....XXX...XXXXXXX.XX.] duration: 5.98992575s trace: https://jaeger.dev.storj.io/trace/32e442c67283be7 FAIL (2025-12-29 03:43:40 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 03:42:30.501741144 (564h13m30.372824179s) [upload=494687]: Starting upload Dec 29 03:42:30.501859240 (564h13m30.372942285s) [upload=494687]: Done starting upload Dec 29 03:42:30.501941336 (564h13m30.373024371s) [upload=494687]: Uploading segments... Dec 29 03:42:30.502099497 (564h13m30.373182552s) [upload=494687]: Got next segment. Inline: false Dec 29 03:42:30.502142750 (564h13m30.373225795s) [upload=494687]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 03:42:30.502148841 (564h13m30.373231876s) [upload=494687]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 03:42:30.746974576 (564h13m30.618057621s) [seg_pos={0 0}, upload=494687]: Begin upload segment... Dec 29 03:42:30.747032586 (564h13m30.618115621s) [seg_pos={0 0}, upload=494687]: Done begin upload segment. Dec 29 03:42:30.747061602 (564h13m30.618144647s) [noise=true , address=145.79.227.11:4731 , piece_id=R35NAF66L573YQJGTGTK3BQZZUF4G4RBW2U5CPMQXNZWAA42COIQ, seg_pos={0 0}, upload=494687]: Uploading piece... Dec 29 03:42:30.747114132 (564h13m30.618197177s) [noise=true , address=77.90.31.14:4661 , piece_id=CTHQGRWKHIBYU627IWDD2RSFB375BEFCT4MU6KV33IWZXUUMUZEA, seg_pos={0 0}, upload=494687]: Uploading piece... Dec 29 03:42:30.747507481 (564h13m30.618590516s) [noise=true , address=145.79.227.15:4874 , piece_id=HOOZ2RSSX7C3ERMZDCI7PCEUKTY25V3FGAHWQATP7BDCAIKTK6UA, seg_pos={0 0}, upload=494687]: Uploading piece... Dec 29 03:42:30.748060845 (564h13m30.61914389s) [noise=true , address=145.79.227.48:4302 , piece_id=5FNSV3R3XN2L6B22BG7WSOQ76HU3OGT6QSEZN6RRSBXMGPAXX72Q, seg_pos={0 0}, upload=494687]: Uploading piece... Dec 29 03:42:30.748436871 (564h13m30.619519916s) [noise=true , address=145.79.227.47:4265 , piece_id=OJY23K6C4FPXPH3LQ7R7DXMB5BCYXKLOFV3LMPX5SGVXQWMQOPBA, seg_pos={0 0}, upload=494687]: Uploading piece... Dec 29 03:42:30.748781086 (564h13m30.619864131s) [noise=true , address=145.79.227.3:4150 , piece_id=FS7FULDIEITCNEKKNHS5OPPE7MZFKKWGAS46FMBSPZREB6FEPOBQ, seg_pos={0 0}, upload=494687]: Uploading piece... Dec 29 03:42:30.749132125 (564h13m30.62021516s) [noise=true , address=145.79.227.50:4373 , piece_id=4DDZV6NKWB7524HG4NP4VDCVXC3YEHVDSB7KIUSPJW7Y2VH3POOA, seg_pos={0 0}, upload=494687]: Uploading piece... Dec 29 03:42:30.749469346 (564h13m30.620552381s) [noise=true , address=45.91.116.5:4298 , piece_id=DPTTNR2CXVB5EDC6WLJUD64JSCPTSKAI62UZXEFGGZRROZRTGI2A, seg_pos={0 0}, upload=494687]: Uploading piece... Dec 29 03:42:30.750809749 (564h13m30.621892784s) [noise=true , address=145.79.227.38:4977 , piece_id=BZGE46XG6SSPQHFV5UHI4HD3M7QAMILYO226SPSMWUJTGR2UBBQA, seg_pos={0 0}, upload=494687]: Uploading piece... Dec 29 03:42:30.751233446 (564h13m30.622316481s) [noise=true , address=145.79.227.42:4121 , piece_id=OGNOTRSW7E6VC3VUUS6VB5C6K5SNY6TW64BOAFBRFYJCVY4F6HMQ, seg_pos={0 0}, upload=494687]: Uploading piece... Dec 29 03:42:30.751743526 (564h13m30.622826571s) [noise=true , address=145.79.227.52:4447 , piece_id=4XMVHAYXFR2RXZSF2DJS4W7MF5AZWRX3M6ZFJGEGJQKM53QDKVGA, seg_pos={0 0}, upload=494687]: Uploading piece... Dec 29 03:42:30.752148728 (564h13m30.623231763s) [noise=true , address=77.90.31.11:4554 , piece_id=6VNUQJ6C7MDP2IE32MOPKLMG2AXMH4LEI4N7QKDHBHV5NPSMNTJA, seg_pos={0 0}, upload=494687]: Uploading piece... Dec 29 03:42:30.753349534 (564h13m30.624432569s) [noise=true , address=45.91.116.6:4975 , piece_id=MMJXX3KC4NQJQGEQPBPZAZKV5L66ST3ZGNOZB5FQN5MVEBNR2RNA, seg_pos={0 0}, upload=494687]: Uploading piece... Dec 29 03:42:30.754606969 (564h13m30.625690004s) [noise=true , address=45.91.116.13:4154 , piece_id=4N3OA7KXCJISXC53CI6IR74T4NVGLYHVR6QRFI62THQPF7XTHGRQ, seg_pos={0 0}, upload=494687]: Uploading piece... Dec 29 03:42:30.756037433 (564h13m30.627120478s) [noise=true , address=145.79.227.12:4767 , piece_id=2PUDZ7TNSYBLPTB6XG4QGGCTWURFPKDM7S2MU22YRTG4FE33P4RA, seg_pos={0 0}, upload=494687]: Uploading piece... Dec 29 03:42:30.757126166 (564h13m30.628209201s) [noise=true , address=145.79.227.5:4221 , piece_id=36MFXYBLO2INBXOSABB2PHOITXCRPSF6J7W2EGDJC2ND3P2LXIWA, seg_pos={0 0}, upload=494687]: Uploading piece... Dec 29 03:42:32.457137751 (564h13m32.328220796s) [upload=494687]: Next returned nil segment Dec 29 03:42:32.457166857 (564h13m32.328249902s) [upload=494687]: Waiting for error group managing segments... Dec 29 03:42:32.737282136 (564h13m32.608365171s) [noise=true , address=145.79.227.5:4221 , piece_id=36MFXYBLO2INBXOSABB2PHOITXCRPSF6J7W2EGDJC2ND3P2LXIWA, seg_pos={0 0}, upload=494687]: Done uploading piece. err: <nil> Dec 29 03:42:32.737314006 (564h13m32.608397051s) [noise=true , address=145.79.227.12:4767 , piece_id=2PUDZ7TNSYBLPTB6XG4QGGCTWURFPKDM7S2MU22YRTG4FE33P4RA, seg_pos={0 0}, upload=494687]: Done uploading piece. err: <nil> Dec 29 03:42:32.745027324 (564h13m32.616110369s) [noise=true , address=145.79.227.11:4731 , piece_id=R35NAF66L573YQJGTGTK3BQZZUF4G4RBW2U5CPMQXNZWAA42COIQ, seg_pos={0 0}, upload=494687]: Done uploading piece. err: <nil> Dec 29 03:42:32.745424670 (564h13m32.616507705s) [noise=true , address=145.79.227.15:4874 , piece_id=HOOZ2RSSX7C3ERMZDCI7PCEUKTY25V3FGAHWQATP7BDCAIKTK6UA, seg_pos={0 0}, upload=494687]: Done uploading piece. err: <nil> Dec 29 03:42:32.746696423 (564h13m32.617779458s) [noise=true , address=145.79.227.52:4447 , piece_id=4XMVHAYXFR2RXZSF2DJS4W7MF5AZWRX3M6ZFJGEGJQKM53QDKVGA, seg_pos={0 0}, upload=494687]: Done uploading piece. err: <nil> Dec 29 03:42:32.746741949 (564h13m32.617824994s) [noise=true , address=145.79.227.50:4373 , piece_id=4DDZV6NKWB7524HG4NP4VDCVXC3YEHVDSB7KIUSPJW7Y2VH3POOA, seg_pos={0 0}, upload=494687]: Done uploading piece. err: <nil> Dec 29 03:42:32.747724349 (564h13m32.618807394s) [noise=true , address=145.79.227.48:4302 , piece_id=5FNSV3R3XN2L6B22BG7WSOQ76HU3OGT6QSEZN6RRSBXMGPAXX72Q, seg_pos={0 0}, upload=494687]: Done uploading piece. err: <nil> Dec 29 03:42:32.754753554 (564h13m32.625836599s) [noise=true , address=145.79.227.47:4265 , piece_id=OJY23K6C4FPXPH3LQ7R7DXMB5BCYXKLOFV3LMPX5SGVXQWMQOPBA, seg_pos={0 0}, upload=494687]: Done uploading piece. err: <nil> Dec 29 03:42:32.766152089 (564h13m32.637235124s) [noise=true , address=145.79.227.42:4121 , piece_id=OGNOTRSW7E6VC3VUUS6VB5C6K5SNY6TW64BOAFBRFYJCVY4F6HMQ, seg_pos={0 0}, upload=494687]: Done uploading piece. err: <nil> Dec 29 03:42:32.768724888 (564h13m32.639807933s) [noise=true , address=145.79.227.38:4977 , piece_id=BZGE46XG6SSPQHFV5UHI4HD3M7QAMILYO226SPSMWUJTGR2UBBQA, seg_pos={0 0}, upload=494687]: Done uploading piece. err: <nil> Dec 29 03:42:32.778979303 (564h13m32.650062338s) [noise=true , address=145.79.227.3:4150 , piece_id=FS7FULDIEITCNEKKNHS5OPPE7MZFKKWGAS46FMBSPZREB6FEPOBQ, seg_pos={0 0}, upload=494687]: Done uploading piece. err: <nil> Dec 29 03:42:32.945069113 (564h13m32.816152158s) [noise=true , address=77.90.31.11:4554 , piece_id=6VNUQJ6C7MDP2IE32MOPKLMG2AXMH4LEI4N7QKDHBHV5NPSMNTJA, seg_pos={0 0}, upload=494687]: Done uploading piece. err: <nil> Dec 29 03:42:32.992546286 (564h13m32.863629331s) [noise=true , address=45.91.116.6:4975 , piece_id=MMJXX3KC4NQJQGEQPBPZAZKV5L66ST3ZGNOZB5FQN5MVEBNR2RNA, seg_pos={0 0}, upload=494687]: Done uploading piece. err: <nil> Dec 29 03:42:32.992571684 (564h13m32.863654719s) [seg_pos={0 0}, upload=494687]: Segment reached optimal threshold of 13 pieces. Dec 29 03:42:32.992854903 (564h13m32.863937979s) [noise=true , address=77.90.31.14:4661 , piece_id=CTHQGRWKHIBYU627IWDD2RSFB375BEFCT4MU6KV33IWZXUUMUZEA, seg_pos={0 0}, upload=494687]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: expected piece hash; context canceled; EOF Dec 29 03:42:32.992891944 (564h13m32.863974979s) [noise=true , address=77.90.31.14:4661 , piece_id=CTHQGRWKHIBYU627IWDD2RSFB375BEFCT4MU6KV33IWZXUUMUZEA, seg_pos={0 0}, upload=494687]: Long tail cancellation due to optimal threshold Dec 29 03:42:32.992925618 (564h13m32.864008653s) [noise=true , address=45.91.116.5:4298 , piece_id=DPTTNR2CXVB5EDC6WLJUD64JSCPTSKAI62UZXEFGGZRROZRTGI2A, seg_pos={0 0}, upload=494687]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: write tcp 188.166.225.170:50518->45.91.116.5:4298: use of closed network connection; write tcp 188.166.225.170:50518->45.91.116.5:4298: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50518->45.91.116.5:4298: use of closed network connection Dec 29 03:42:32.992941758 (564h13m32.864024793s) [noise=true , address=45.91.116.5:4298 , piece_id=DPTTNR2CXVB5EDC6WLJUD64JSCPTSKAI62UZXEFGGZRROZRTGI2A, seg_pos={0 0}, upload=494687]: Long tail cancellation due to optimal threshold Dec 29 03:42:32.993087196 (564h13m32.864170241s) [noise=true , address=45.91.116.13:4154 , piece_id=4N3OA7KXCJISXC53CI6IR74T4NVGLYHVR6QRFI62THQPF7XTHGRQ, seg_pos={0 0}, upload=494687]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN): protocol: expected piece hash; context canceled; EOF Dec 29 03:42:32.993093798 (564h13m32.864176834s) [noise=true , address=45.91.116.13:4154 , piece_id=4N3OA7KXCJISXC53CI6IR74T4NVGLYHVR6QRFI62THQPF7XTHGRQ, seg_pos={0 0}, upload=494687]: Long tail cancellation due to optimal threshold Dec 29 03:42:32.993119818 (564h13m32.864202853s) [seg_pos={0 0}, upload=494687]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 03:42:32.993209308 (564h13m32.864292353s) [upload=494687]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 03:42:32.993214037 (564h13m32.864297072s) [upload=494687]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 03:42:33.296911160 (564h13m33.167994205s) [upload=494687]: Done uploading segments... Dec 29 03:42:33.296956156 (564h13m33.168039191s) [upload=494687]: Upload finished. err: <nil> Duration: 5.989939346s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..................................X...................X.....] duration: 51.30403618s trace: https://jaeger.dev.storj.io/trace/2408171d336afafd FAIL (2025-12-29 03:43:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12Tz3HEHACmHU9Gp2W2vHVuF5PXNmdsa5N8nYhwt52HCTjggqRD, Piece ID: S3MEER2QCVCORYQNYNY2S22TC2H5HHKZW7VQJ54VL754UZTD42JQ) context deadline exceeded; piecestore: (Node ID: 1ve7KKYDbNo8Ma7wGVBH5x9z7zbPtPAPztmuQ9r6ijPpVgh5NX, Piece ID: S7T55BAWP53YJVHPW2GPLUM2VB3NBHZ2YD4UWBKKWCFHDAXAW3KQ) context deadline exceeded; piecestore: (Node ID: 1WVHPacX35JJ8PFzCs3UtGwsR4yVHXKmiqfGedK9KfXbHNJtmu, Piece ID: NYWKOTPLBZQIHGSQFMTT2QE6OQX7H5OKFVWJK3I65CWGGDSDDBMA) context deadline exceeded ======LOGS====== upload=494622]: Uploading piece... Dec 29 03:37:43.315384541 (564h8m43.186467596s) [noise=true , address=145.79.227.11:4762 , piece_id=2GQ5ASSBMUC52NCYRLBF3EOKYEA3G6DTIA6K56DE3OUQ4F5OLX2Q, seg_pos={0 0}, upload=494622]: Uploading piece... Dec 29 03:37:43.315899872 (564h8m43.186982907s) [noise=true , address=77.90.31.13:4649 , piece_id=S7T55BAWP53YJVHPW2GPLUM2VB3NBHZ2YD4UWBKKWCFHDAXAW3KQ, seg_pos={0 0}, upload=494622]: Uploading piece... Dec 29 03:37:43.316395165 (564h8m43.1874782s) [noise=true , address=77.90.31.18:4834 , piece_id=FQS6BFFJTORJPKJCINDZKXTY6GFRQTR3FZPWAEG7F2DNROUEPGJA, seg_pos={0 0}, upload=494622]: Uploading piece... Dec 29 03:37:43.316885961 (564h8m43.187969006s) [noise=true , address=193.149.164.143:4695 , piece_id=YF4EWRMS3GWEYPSA67GLG6EA5H7GZ55GTA3ISIEJ7RFEG2NDTLEQ, seg_pos={0 0}, upload=494622]: Uploading piece... Dec 29 03:37:43.317355795 (564h8m43.18843883s) [noise=true , address=145.79.227.7:4297 , piece_id=NNJ2KDDLNZHT4SGOUXBWBYD5OGW6MTGCAWZ63D5LH7BGM6KOJGDA, seg_pos={0 0}, upload=494622]: Uploading piece... Dec 29 03:37:43.317364631 (564h8m43.188447666s) [noise=true , address=145.79.227.33:4886 , piece_id=DB64UXSDC47SRTQX7PJGE66RCL7XFW3TGMCU3JOWVEJVGVHLYGWQ, seg_pos={0 0}, upload=494622]: Uploading piece... Dec 29 03:37:43.317689451 (564h8m43.188772486s) [noise=true , address=64.49.13.26:4950 , piece_id=HQ6BBIFQAD5RRN7FRWGDFHAVVMSDUWJUGTX7LBTA2TQJ3PNWKUHA, seg_pos={0 0}, upload=494622]: Uploading piece... Dec 29 03:37:43.317877448 (564h8m43.188960493s) [noise=true , address=45.91.116.10:4096 , piece_id=VIQK6EV67DONFZY77BOTWHY7POJEOE7FDHB6TKQMHVSHJJM35V7Q, seg_pos={0 0}, upload=494622]: Uploading piece... Dec 29 03:37:43.318315813 (564h8m43.189398858s) [noise=true , address=145.79.227.9:4388 , piece_id=DQU5TSAWTEEIBT272I76ZMGXD6FJT2QMXQISRDMWG5NOPORLBLLA, seg_pos={0 0}, upload=494622]: Uploading piece... Dec 29 03:37:43.318722317 (564h8m43.189805352s) [noise=true , address=77.90.31.12:4600 , piece_id=N4KXD5SYPYOKBXQRSL3UKNDDV7XB4KOKRYRYKGMMAHMAO6VVOEHQ, seg_pos={0 0}, upload=494622]: Uploading piece... Dec 29 03:37:43.319195248 (564h8m43.190278293s) [noise=true , address=45.91.116.8:4050 , piece_id=NYWKOTPLBZQIHGSQFMTT2QE6OQX7H5OKFVWJK3I65CWGGDSDDBMA, seg_pos={0 0}, upload=494622]: Uploading piece... Dec 29 03:37:43.319666174 (564h8m43.190749219s) [noise=true , address=77.90.31.21:4899 , piece_id=4MZKL4KVJF4DYGVPYGVOUHD6TYIDCRER33YPW6RG7KT6B6IXQEOA, seg_pos={0 0}, upload=494622]: Uploading piece... Dec 29 03:37:43.321557184 (564h8m43.192640229s) [noise=true , address=145.79.227.34:4909 , piece_id=UCGJPN357USFHIQ7II5GLRKNDE4H4HUVDQ7CB5R27C3VGRP2R3IA, seg_pos={0 0}, upload=494622]: Uploading piece... Dec 29 03:37:43.322231088 (564h8m43.193314133s) [noise=true , address=45.91.116.11:4139 , piece_id=S3MEER2QCVCORYQNYNY2S22TC2H5HHKZW7VQJ54VL754UZTD42JQ, seg_pos={0 0}, upload=494622]: Uploading piece... Dec 29 03:37:43.322851720 (564h8m43.193934755s) [noise=true , address=145.79.227.51:4444 , piece_id=EWEXXYYWF43J7FG6NPSJ2PICHTRPY533L4DQQX2CSW5IMJ6CB2PQ, seg_pos={0 0}, upload=494622]: Uploading piece... Dec 29 03:37:43.323435972 (564h8m43.194519007s) [noise=true , address=145.79.227.40:4068 , piece_id=4QOJWRVEE3ERM2AHP764XSTE7ZYHUUJ2ENBVXRS5UDIGKZOK7YKA, seg_pos={0 0}, upload=494622]: Uploading piece... Dec 29 03:37:43.323901118 (564h8m43.194984153s) [noise=true , address=145.79.227.30:4790 , piece_id=2TSIHAVWOZPOVKSTG2ZXBBEFEMMMRLMOWHM47TDANARFPDJM3NPQ, seg_pos={0 0}, upload=494622]: Uploading piece... Dec 29 03:37:44.620573279 (564h8m44.491656314s) [upload=494622]: Next returned nil segment Dec 29 03:37:44.620593617 (564h8m44.491676652s) [upload=494622]: Waiting for error group managing segments... Dec 29 03:37:45.178264102 (564h8m45.049347137s) [noise=true , address=145.79.227.11:4762 , piece_id=2GQ5ASSBMUC52NCYRLBF3EOKYEA3G6DTIA6K56DE3OUQ4F5OLX2Q, seg_pos={0 0}, upload=494622]: Done uploading piece. err: <nil> Dec 29 03:37:45.190189951 (564h8m45.061272986s) [noise=true , address=145.79.227.51:4444 , piece_id=EWEXXYYWF43J7FG6NPSJ2PICHTRPY533L4DQQX2CSW5IMJ6CB2PQ, seg_pos={0 0}, upload=494622]: Done uploading piece. err: <nil> Dec 29 03:37:45.199163818 (564h8m45.070246853s) [noise=true , address=145.79.227.7:4297 , piece_id=NNJ2KDDLNZHT4SGOUXBWBYD5OGW6MTGCAWZ63D5LH7BGM6KOJGDA, seg_pos={0 0}, upload=494622]: Done uploading piece. err: <nil> Dec 29 03:37:45.448727893 (564h8m45.319810928s) [noise=true , address=45.91.116.11:4139 , piece_id=S3MEER2QCVCORYQNYNY2S22TC2H5HHKZW7VQJ54VL754UZTD42JQ, seg_pos={0 0}, upload=494622]: Done uploading piece. err: <nil> Dec 29 03:37:45.459119140 (564h8m45.330202175s) [noise=true , address=77.90.31.13:4649 , piece_id=S7T55BAWP53YJVHPW2GPLUM2VB3NBHZ2YD4UWBKKWCFHDAXAW3KQ, seg_pos={0 0}, upload=494622]: Done uploading piece. err: <nil> Dec 29 03:37:45.471609714 (564h8m45.342692759s) [noise=true , address=45.91.116.8:4050 , piece_id=NYWKOTPLBZQIHGSQFMTT2QE6OQX7H5OKFVWJK3I65CWGGDSDDBMA, seg_pos={0 0}, upload=494622]: Done uploading piece. err: <nil> Dec 29 03:37:45.489098314 (564h8m45.36018136s) [noise=true , address=77.90.31.21:4899 , piece_id=4MZKL4KVJF4DYGVPYGVOUHD6TYIDCRER33YPW6RG7KT6B6IXQEOA, seg_pos={0 0}, upload=494622]: Done uploading piece. err: <nil> Dec 29 03:37:45.489726060 (564h8m45.360809105s) [noise=true , address=77.90.31.11:4574 , piece_id=3TP2ME5NTXVK45B34W5U4DESLFJLOJEPRQOJLEEPN2PIMYS2IDWQ, seg_pos={0 0}, upload=494622]: Done uploading piece. err: <nil> Dec 29 03:37:45.779669865 (564h8m45.65075291s) [noise=true , address=145.79.227.9:4388 , piece_id=DQU5TSAWTEEIBT272I76ZMGXD6FJT2QMXQISRDMWG5NOPORLBLLA, seg_pos={0 0}, upload=494622]: Done uploading piece. err: <nil> Dec 29 03:37:45.817564973 (564h8m45.688648008s) [noise=true , address=77.90.31.18:4834 , piece_id=FQS6BFFJTORJPKJCINDZKXTY6GFRQTR3FZPWAEG7F2DNROUEPGJA, seg_pos={0 0}, upload=494622]: Done uploading piece. err: <nil> Dec 29 03:37:46.001090802 (564h8m45.872173857s) [noise=true , address=193.149.164.143:4695 , piece_id=YF4EWRMS3GWEYPSA67GLG6EA5H7GZ55GTA3ISIEJ7RFEG2NDTLEQ, seg_pos={0 0}, upload=494622]: Done uploading piece. err: <nil> Dec 29 03:37:46.129753764 (564h8m46.000836819s) [noise=true , address=145.79.227.33:4886 , piece_id=DB64UXSDC47SRTQX7PJGE66RCL7XFW3TGMCU3JOWVEJVGVHLYGWQ, seg_pos={0 0}, upload=494622]: Done uploading piece. err: <nil> Dec 29 03:37:46.163280064 (564h8m46.034363109s) [noise=true , address=145.79.227.34:4909 , piece_id=UCGJPN357USFHIQ7II5GLRKNDE4H4HUVDQ7CB5R27C3VGRP2R3IA, seg_pos={0 0}, upload=494622]: Done uploading piece. err: <nil> Dec 29 03:37:46.167742170 (564h8m46.038825205s) [noise=true , address=145.79.227.30:4790 , piece_id=2TSIHAVWOZPOVKSTG2ZXBBEFEMMMRLMOWHM47TDANARFPDJM3NPQ, seg_pos={0 0}, upload=494622]: Done uploading piece. err: <nil> Dec 29 03:37:46.167764403 (564h8m46.038847448s) [seg_pos={0 0}, upload=494622]: Segment reached optimal threshold of 14 pieces. Dec 29 03:37:46.167996534 (564h8m46.039079569s) [noise=true , address=45.91.116.10:4096 , piece_id=VIQK6EV67DONFZY77BOTWHY7POJEOE7FDHB6TKQMHVSHJJM35V7Q, seg_pos={0 0}, upload=494622]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12MZX1GwZGrEi2q44dPiYrwMoum5A2caZtXTS3qiLNcoyHhon7p): protocol: expected piece hash; context canceled; EOF Dec 29 03:37:46.168005812 (564h8m46.039088857s) [noise=true , address=45.91.116.10:4096 , piece_id=VIQK6EV67DONFZY77BOTWHY7POJEOE7FDHB6TKQMHVSHJJM35V7Q, seg_pos={0 0}, upload=494622]: Long tail cancellation due to optimal threshold Dec 29 03:37:46.168120160 (564h8m46.039203195s) [noise=true , address=64.49.13.26:4950 , piece_id=HQ6BBIFQAD5RRN7FRWGDFHAVVMSDUWJUGTX7LBTA2TQJ3PNWKUHA, seg_pos={0 0}, upload=494622]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14Tv25fLSqTnZQiDzrYqKCSbdt6ep3upgqD2Se8QSRuQFynkSN): protocol: expected piece hash; context canceled; EOF Dec 29 03:37:46.168136090 (564h8m46.039219125s) [noise=true , address=64.49.13.26:4950 , piece_id=HQ6BBIFQAD5RRN7FRWGDFHAVVMSDUWJUGTX7LBTA2TQJ3PNWKUHA, seg_pos={0 0}, upload=494622]: Long tail cancellation due to optimal threshold Dec 29 03:37:46.168274995 (564h8m46.03935804s) [noise=true , address=77.90.31.12:4600 , piece_id=N4KXD5SYPYOKBXQRSL3UKNDDV7XB4KOKRYRYKGMMAHMAO6VVOEHQ, seg_pos={0 0}, upload=494622]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PKT2EcU6jAe6oUUdiTcwxGQsFa36uEmown77SemqQMM5Xw5nc): protocol: expected piece hash; context canceled; EOF Dec 29 03:37:46.168283491 (564h8m46.039366536s) [noise=true , address=77.90.31.12:4600 , piece_id=N4KXD5SYPYOKBXQRSL3UKNDDV7XB4KOKRYRYKGMMAHMAO6VVOEHQ, seg_pos={0 0}, upload=494622]: Long tail cancellation due to optimal threshold Dec 29 03:37:46.168394032 (564h8m46.039477067s) [noise=true , address=145.79.227.40:4068 , piece_id=4QOJWRVEE3ERM2AHP764XSTE7ZYHUUJ2ENBVXRS5UDIGKZOK7YKA, seg_pos={0 0}, upload=494622]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Lh9dYpHCxbs7gpZ4Xt7os5Gm5Nco6jrNW1V1FHddVUEMkFfPY): protocol: expected piece hash; context canceled; EOF Dec 29 03:37:46.168403810 (564h8m46.039486845s) [noise=true , address=145.79.227.40:4068 , piece_id=4QOJWRVEE3ERM2AHP764XSTE7ZYHUUJ2ENBVXRS5UDIGKZOK7YKA, seg_pos={0 0}, upload=494622]: Long tail cancellation due to optimal threshold Dec 29 03:37:46.168432725 (564h8m46.03951577s) [seg_pos={0 0}, upload=494622]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 03:37:46.168576960 (564h8m46.039659995s) [upload=494622]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 03:37:46.168584033 (564h8m46.039667068s) [upload=494622]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 03:37:46.466683658 (564h8m46.337766723s) [upload=494622]: Done uploading segments... Dec 29 03:37:46.466712052 (564h8m46.337795087s) [upload=494622]: Upload finished. err: <nil> Duration: 51.304069353s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [............................XX................X......XXX....] duration: 3.496150351s trace: https://jaeger.dev.storj.io/trace/79e92355ce966062 FAIL (2025-12-29 03:43:14 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.496177853s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 30 out of 60 is failed (threshold=3) [XX.....XX.X.....X.X..XXXX..XXX.XX...X.X.....XXX...XXXXXXX.XX] duration: 30.002498622s trace: https://jaeger.dev.storj.io/trace/32e442c67283be7 FAIL (2025-12-29 03:42:59 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 03:42:30.501741144 (564h13m30.372824179s) [upload=494687]: Starting upload Dec 29 03:42:30.501859240 (564h13m30.372942285s) [upload=494687]: Done starting upload Dec 29 03:42:30.501941336 (564h13m30.373024371s) [upload=494687]: Uploading segments... Dec 29 03:42:30.502099497 (564h13m30.373182552s) [upload=494687]: Got next segment. Inline: false Dec 29 03:42:30.502142750 (564h13m30.373225795s) [upload=494687]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 03:42:30.502148841 (564h13m30.373231876s) [upload=494687]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 03:42:30.746974576 (564h13m30.618057621s) [seg_pos={0 0}, upload=494687]: Begin upload segment... Dec 29 03:42:30.747032586 (564h13m30.618115621s) [seg_pos={0 0}, upload=494687]: Done begin upload segment. Dec 29 03:42:30.747061602 (564h13m30.618144647s) [noise=true , address=145.79.227.11:4731 , piece_id=R35NAF66L573YQJGTGTK3BQZZUF4G4RBW2U5CPMQXNZWAA42COIQ, seg_pos={0 0}, upload=494687]: Uploading piece... Dec 29 03:42:30.747114132 (564h13m30.618197177s) [noise=true , address=77.90.31.14:4661 , piece_id=CTHQGRWKHIBYU627IWDD2RSFB375BEFCT4MU6KV33IWZXUUMUZEA, seg_pos={0 0}, upload=494687]: Uploading piece... Dec 29 03:42:30.747507481 (564h13m30.618590516s) [noise=true , address=145.79.227.15:4874 , piece_id=HOOZ2RSSX7C3ERMZDCI7PCEUKTY25V3FGAHWQATP7BDCAIKTK6UA, seg_pos={0 0}, upload=494687]: Uploading piece... Dec 29 03:42:30.748060845 (564h13m30.61914389s) [noise=true , address=145.79.227.48:4302 , piece_id=5FNSV3R3XN2L6B22BG7WSOQ76HU3OGT6QSEZN6RRSBXMGPAXX72Q, seg_pos={0 0}, upload=494687]: Uploading piece... Dec 29 03:42:30.748436871 (564h13m30.619519916s) [noise=true , address=145.79.227.47:4265 , piece_id=OJY23K6C4FPXPH3LQ7R7DXMB5BCYXKLOFV3LMPX5SGVXQWMQOPBA, seg_pos={0 0}, upload=494687]: Uploading piece... Dec 29 03:42:30.748781086 (564h13m30.619864131s) [noise=true , address=145.79.227.3:4150 , piece_id=FS7FULDIEITCNEKKNHS5OPPE7MZFKKWGAS46FMBSPZREB6FEPOBQ, seg_pos={0 0}, upload=494687]: Uploading piece... Dec 29 03:42:30.749132125 (564h13m30.62021516s) [noise=true , address=145.79.227.50:4373 , piece_id=4DDZV6NKWB7524HG4NP4VDCVXC3YEHVDSB7KIUSPJW7Y2VH3POOA, seg_pos={0 0}, upload=494687]: Uploading piece... Dec 29 03:42:30.749469346 (564h13m30.620552381s) [noise=true , address=45.91.116.5:4298 , piece_id=DPTTNR2CXVB5EDC6WLJUD64JSCPTSKAI62UZXEFGGZRROZRTGI2A, seg_pos={0 0}, upload=494687]: Uploading piece... Dec 29 03:42:30.750809749 (564h13m30.621892784s) [noise=true , address=145.79.227.38:4977 , piece_id=BZGE46XG6SSPQHFV5UHI4HD3M7QAMILYO226SPSMWUJTGR2UBBQA, seg_pos={0 0}, upload=494687]: Uploading piece... Dec 29 03:42:30.751233446 (564h13m30.622316481s) [noise=true , address=145.79.227.42:4121 , piece_id=OGNOTRSW7E6VC3VUUS6VB5C6K5SNY6TW64BOAFBRFYJCVY4F6HMQ, seg_pos={0 0}, upload=494687]: Uploading piece... Dec 29 03:42:30.751743526 (564h13m30.622826571s) [noise=true , address=145.79.227.52:4447 , piece_id=4XMVHAYXFR2RXZSF2DJS4W7MF5AZWRX3M6ZFJGEGJQKM53QDKVGA, seg_pos={0 0}, upload=494687]: Uploading piece... Dec 29 03:42:30.752148728 (564h13m30.623231763s) [noise=true , address=77.90.31.11:4554 , piece_id=6VNUQJ6C7MDP2IE32MOPKLMG2AXMH4LEI4N7QKDHBHV5NPSMNTJA, seg_pos={0 0}, upload=494687]: Uploading piece... Dec 29 03:42:30.753349534 (564h13m30.624432569s) [noise=true , address=45.91.116.6:4975 , piece_id=MMJXX3KC4NQJQGEQPBPZAZKV5L66ST3ZGNOZB5FQN5MVEBNR2RNA, seg_pos={0 0}, upload=494687]: Uploading piece... Dec 29 03:42:30.754606969 (564h13m30.625690004s) [noise=true , address=45.91.116.13:4154 , piece_id=4N3OA7KXCJISXC53CI6IR74T4NVGLYHVR6QRFI62THQPF7XTHGRQ, seg_pos={0 0}, upload=494687]: Uploading piece... Dec 29 03:42:30.756037433 (564h13m30.627120478s) [noise=true , address=145.79.227.12:4767 , piece_id=2PUDZ7TNSYBLPTB6XG4QGGCTWURFPKDM7S2MU22YRTG4FE33P4RA, seg_pos={0 0}, upload=494687]: Uploading piece... Dec 29 03:42:30.757126166 (564h13m30.628209201s) [noise=true , address=145.79.227.5:4221 , piece_id=36MFXYBLO2INBXOSABB2PHOITXCRPSF6J7W2EGDJC2ND3P2LXIWA, seg_pos={0 0}, upload=494687]: Uploading piece... Dec 29 03:42:32.457137751 (564h13m32.328220796s) [upload=494687]: Next returned nil segment Dec 29 03:42:32.457166857 (564h13m32.328249902s) [upload=494687]: Waiting for error group managing segments... Dec 29 03:42:32.737282136 (564h13m32.608365171s) [noise=true , address=145.79.227.5:4221 , piece_id=36MFXYBLO2INBXOSABB2PHOITXCRPSF6J7W2EGDJC2ND3P2LXIWA, seg_pos={0 0}, upload=494687]: Done uploading piece. err: <nil> Dec 29 03:42:32.737314006 (564h13m32.608397051s) [noise=true , address=145.79.227.12:4767 , piece_id=2PUDZ7TNSYBLPTB6XG4QGGCTWURFPKDM7S2MU22YRTG4FE33P4RA, seg_pos={0 0}, upload=494687]: Done uploading piece. err: <nil> Dec 29 03:42:32.745027324 (564h13m32.616110369s) [noise=true , address=145.79.227.11:4731 , piece_id=R35NAF66L573YQJGTGTK3BQZZUF4G4RBW2U5CPMQXNZWAA42COIQ, seg_pos={0 0}, upload=494687]: Done uploading piece. err: <nil> Dec 29 03:42:32.745424670 (564h13m32.616507705s) [noise=true , address=145.79.227.15:4874 , piece_id=HOOZ2RSSX7C3ERMZDCI7PCEUKTY25V3FGAHWQATP7BDCAIKTK6UA, seg_pos={0 0}, upload=494687]: Done uploading piece. err: <nil> Dec 29 03:42:32.746696423 (564h13m32.617779458s) [noise=true , address=145.79.227.52:4447 , piece_id=4XMVHAYXFR2RXZSF2DJS4W7MF5AZWRX3M6ZFJGEGJQKM53QDKVGA, seg_pos={0 0}, upload=494687]: Done uploading piece. err: <nil> Dec 29 03:42:32.746741949 (564h13m32.617824994s) [noise=true , address=145.79.227.50:4373 , piece_id=4DDZV6NKWB7524HG4NP4VDCVXC3YEHVDSB7KIUSPJW7Y2VH3POOA, seg_pos={0 0}, upload=494687]: Done uploading piece. err: <nil> Dec 29 03:42:32.747724349 (564h13m32.618807394s) [noise=true , address=145.79.227.48:4302 , piece_id=5FNSV3R3XN2L6B22BG7WSOQ76HU3OGT6QSEZN6RRSBXMGPAXX72Q, seg_pos={0 0}, upload=494687]: Done uploading piece. err: <nil> Dec 29 03:42:32.754753554 (564h13m32.625836599s) [noise=true , address=145.79.227.47:4265 , piece_id=OJY23K6C4FPXPH3LQ7R7DXMB5BCYXKLOFV3LMPX5SGVXQWMQOPBA, seg_pos={0 0}, upload=494687]: Done uploading piece. err: <nil> Dec 29 03:42:32.766152089 (564h13m32.637235124s) [noise=true , address=145.79.227.42:4121 , piece_id=OGNOTRSW7E6VC3VUUS6VB5C6K5SNY6TW64BOAFBRFYJCVY4F6HMQ, seg_pos={0 0}, upload=494687]: Done uploading piece. err: <nil> Dec 29 03:42:32.768724888 (564h13m32.639807933s) [noise=true , address=145.79.227.38:4977 , piece_id=BZGE46XG6SSPQHFV5UHI4HD3M7QAMILYO226SPSMWUJTGR2UBBQA, seg_pos={0 0}, upload=494687]: Done uploading piece. err: <nil> Dec 29 03:42:32.778979303 (564h13m32.650062338s) [noise=true , address=145.79.227.3:4150 , piece_id=FS7FULDIEITCNEKKNHS5OPPE7MZFKKWGAS46FMBSPZREB6FEPOBQ, seg_pos={0 0}, upload=494687]: Done uploading piece. err: <nil> Dec 29 03:42:32.945069113 (564h13m32.816152158s) [noise=true , address=77.90.31.11:4554 , piece_id=6VNUQJ6C7MDP2IE32MOPKLMG2AXMH4LEI4N7QKDHBHV5NPSMNTJA, seg_pos={0 0}, upload=494687]: Done uploading piece. err: <nil> Dec 29 03:42:32.992546286 (564h13m32.863629331s) [noise=true , address=45.91.116.6:4975 , piece_id=MMJXX3KC4NQJQGEQPBPZAZKV5L66ST3ZGNOZB5FQN5MVEBNR2RNA, seg_pos={0 0}, upload=494687]: Done uploading piece. err: <nil> Dec 29 03:42:32.992571684 (564h13m32.863654719s) [seg_pos={0 0}, upload=494687]: Segment reached optimal threshold of 13 pieces. Dec 29 03:42:32.992854903 (564h13m32.863937979s) [noise=true , address=77.90.31.14:4661 , piece_id=CTHQGRWKHIBYU627IWDD2RSFB375BEFCT4MU6KV33IWZXUUMUZEA, seg_pos={0 0}, upload=494687]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: expected piece hash; context canceled; EOF Dec 29 03:42:32.992891944 (564h13m32.863974979s) [noise=true , address=77.90.31.14:4661 , piece_id=CTHQGRWKHIBYU627IWDD2RSFB375BEFCT4MU6KV33IWZXUUMUZEA, seg_pos={0 0}, upload=494687]: Long tail cancellation due to optimal threshold Dec 29 03:42:32.992925618 (564h13m32.864008653s) [noise=true , address=45.91.116.5:4298 , piece_id=DPTTNR2CXVB5EDC6WLJUD64JSCPTSKAI62UZXEFGGZRROZRTGI2A, seg_pos={0 0}, upload=494687]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: write tcp 188.166.225.170:50518->45.91.116.5:4298: use of closed network connection; write tcp 188.166.225.170:50518->45.91.116.5:4298: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50518->45.91.116.5:4298: use of closed network connection Dec 29 03:42:32.992941758 (564h13m32.864024793s) [noise=true , address=45.91.116.5:4298 , piece_id=DPTTNR2CXVB5EDC6WLJUD64JSCPTSKAI62UZXEFGGZRROZRTGI2A, seg_pos={0 0}, upload=494687]: Long tail cancellation due to optimal threshold Dec 29 03:42:32.993087196 (564h13m32.864170241s) [noise=true , address=45.91.116.13:4154 , piece_id=4N3OA7KXCJISXC53CI6IR74T4NVGLYHVR6QRFI62THQPF7XTHGRQ, seg_pos={0 0}, upload=494687]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN): protocol: expected piece hash; context canceled; EOF Dec 29 03:42:32.993093798 (564h13m32.864176834s) [noise=true , address=45.91.116.13:4154 , piece_id=4N3OA7KXCJISXC53CI6IR74T4NVGLYHVR6QRFI62THQPF7XTHGRQ, seg_pos={0 0}, upload=494687]: Long tail cancellation due to optimal threshold Dec 29 03:42:32.993119818 (564h13m32.864202853s) [seg_pos={0 0}, upload=494687]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 03:42:32.993209308 (564h13m32.864292353s) [upload=494687]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 03:42:32.993214037 (564h13m32.864297072s) [upload=494687]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 03:42:33.296911160 (564h13m33.167994205s) [upload=494687]: Done uploading segments... Dec 29 03:42:33.296956156 (564h13m33.168039191s) [upload=494687]: Upload finished. err: <nil> Duration: 30.002519952s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [.............................XX................X......XXX...] duration: 5.001297878s trace: https://jaeger.dev.storj.io/trace/79e92355ce966062 FAIL (2025-12-29 03:42:36 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001321803s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [...................................X...................X....] duration: 54.307990664s trace: https://jaeger.dev.storj.io/trace/2408171d336afafd FAIL (2025-12-29 03:42:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12Tz3HEHACmHU9Gp2W2vHVuF5PXNmdsa5N8nYhwt52HCTjggqRD, Piece ID: S3MEER2QCVCORYQNYNY2S22TC2H5HHKZW7VQJ54VL754UZTD42JQ) context deadline exceeded; piecestore: (Node ID: 1ve7KKYDbNo8Ma7wGVBH5x9z7zbPtPAPztmuQ9r6ijPpVgh5NX, Piece ID: S7T55BAWP53YJVHPW2GPLUM2VB3NBHZ2YD4UWBKKWCFHDAXAW3KQ) context deadline exceeded; piecestore: (Node ID: 1WVHPacX35JJ8PFzCs3UtGwsR4yVHXKmiqfGedK9KfXbHNJtmu, Piece ID: NYWKOTPLBZQIHGSQFMTT2QE6OQX7H5OKFVWJK3I65CWGGDSDDBMA) context deadline exceeded ======LOGS====== upload=494622]: Uploading piece... Dec 29 03:37:43.315384541 (564h8m43.186467596s) [noise=true , address=145.79.227.11:4762 , piece_id=2GQ5ASSBMUC52NCYRLBF3EOKYEA3G6DTIA6K56DE3OUQ4F5OLX2Q, seg_pos={0 0}, upload=494622]: Uploading piece... Dec 29 03:37:43.315899872 (564h8m43.186982907s) [noise=true , address=77.90.31.13:4649 , piece_id=S7T55BAWP53YJVHPW2GPLUM2VB3NBHZ2YD4UWBKKWCFHDAXAW3KQ, seg_pos={0 0}, upload=494622]: Uploading piece... Dec 29 03:37:43.316395165 (564h8m43.1874782s) [noise=true , address=77.90.31.18:4834 , piece_id=FQS6BFFJTORJPKJCINDZKXTY6GFRQTR3FZPWAEG7F2DNROUEPGJA, seg_pos={0 0}, upload=494622]: Uploading piece... Dec 29 03:37:43.316885961 (564h8m43.187969006s) [noise=true , address=193.149.164.143:4695 , piece_id=YF4EWRMS3GWEYPSA67GLG6EA5H7GZ55GTA3ISIEJ7RFEG2NDTLEQ, seg_pos={0 0}, upload=494622]: Uploading piece... Dec 29 03:37:43.317355795 (564h8m43.18843883s) [noise=true , address=145.79.227.7:4297 , piece_id=NNJ2KDDLNZHT4SGOUXBWBYD5OGW6MTGCAWZ63D5LH7BGM6KOJGDA, seg_pos={0 0}, upload=494622]: Uploading piece... Dec 29 03:37:43.317364631 (564h8m43.188447666s) [noise=true , address=145.79.227.33:4886 , piece_id=DB64UXSDC47SRTQX7PJGE66RCL7XFW3TGMCU3JOWVEJVGVHLYGWQ, seg_pos={0 0}, upload=494622]: Uploading piece... Dec 29 03:37:43.317689451 (564h8m43.188772486s) [noise=true , address=64.49.13.26:4950 , piece_id=HQ6BBIFQAD5RRN7FRWGDFHAVVMSDUWJUGTX7LBTA2TQJ3PNWKUHA, seg_pos={0 0}, upload=494622]: Uploading piece... Dec 29 03:37:43.317877448 (564h8m43.188960493s) [noise=true , address=45.91.116.10:4096 , piece_id=VIQK6EV67DONFZY77BOTWHY7POJEOE7FDHB6TKQMHVSHJJM35V7Q, seg_pos={0 0}, upload=494622]: Uploading piece... Dec 29 03:37:43.318315813 (564h8m43.189398858s) [noise=true , address=145.79.227.9:4388 , piece_id=DQU5TSAWTEEIBT272I76ZMGXD6FJT2QMXQISRDMWG5NOPORLBLLA, seg_pos={0 0}, upload=494622]: Uploading piece... Dec 29 03:37:43.318722317 (564h8m43.189805352s) [noise=true , address=77.90.31.12:4600 , piece_id=N4KXD5SYPYOKBXQRSL3UKNDDV7XB4KOKRYRYKGMMAHMAO6VVOEHQ, seg_pos={0 0}, upload=494622]: Uploading piece... Dec 29 03:37:43.319195248 (564h8m43.190278293s) [noise=true , address=45.91.116.8:4050 , piece_id=NYWKOTPLBZQIHGSQFMTT2QE6OQX7H5OKFVWJK3I65CWGGDSDDBMA, seg_pos={0 0}, upload=494622]: Uploading piece... Dec 29 03:37:43.319666174 (564h8m43.190749219s) [noise=true , address=77.90.31.21:4899 , piece_id=4MZKL4KVJF4DYGVPYGVOUHD6TYIDCRER33YPW6RG7KT6B6IXQEOA, seg_pos={0 0}, upload=494622]: Uploading piece... Dec 29 03:37:43.321557184 (564h8m43.192640229s) [noise=true , address=145.79.227.34:4909 , piece_id=UCGJPN357USFHIQ7II5GLRKNDE4H4HUVDQ7CB5R27C3VGRP2R3IA, seg_pos={0 0}, upload=494622]: Uploading piece... Dec 29 03:37:43.322231088 (564h8m43.193314133s) [noise=true , address=45.91.116.11:4139 , piece_id=S3MEER2QCVCORYQNYNY2S22TC2H5HHKZW7VQJ54VL754UZTD42JQ, seg_pos={0 0}, upload=494622]: Uploading piece... Dec 29 03:37:43.322851720 (564h8m43.193934755s) [noise=true , address=145.79.227.51:4444 , piece_id=EWEXXYYWF43J7FG6NPSJ2PICHTRPY533L4DQQX2CSW5IMJ6CB2PQ, seg_pos={0 0}, upload=494622]: Uploading piece... Dec 29 03:37:43.323435972 (564h8m43.194519007s) [noise=true , address=145.79.227.40:4068 , piece_id=4QOJWRVEE3ERM2AHP764XSTE7ZYHUUJ2ENBVXRS5UDIGKZOK7YKA, seg_pos={0 0}, upload=494622]: Uploading piece... Dec 29 03:37:43.323901118 (564h8m43.194984153s) [noise=true , address=145.79.227.30:4790 , piece_id=2TSIHAVWOZPOVKSTG2ZXBBEFEMMMRLMOWHM47TDANARFPDJM3NPQ, seg_pos={0 0}, upload=494622]: Uploading piece... Dec 29 03:37:44.620573279 (564h8m44.491656314s) [upload=494622]: Next returned nil segment Dec 29 03:37:44.620593617 (564h8m44.491676652s) [upload=494622]: Waiting for error group managing segments... Dec 29 03:37:45.178264102 (564h8m45.049347137s) [noise=true , address=145.79.227.11:4762 , piece_id=2GQ5ASSBMUC52NCYRLBF3EOKYEA3G6DTIA6K56DE3OUQ4F5OLX2Q, seg_pos={0 0}, upload=494622]: Done uploading piece. err: <nil> Dec 29 03:37:45.190189951 (564h8m45.061272986s) [noise=true , address=145.79.227.51:4444 , piece_id=EWEXXYYWF43J7FG6NPSJ2PICHTRPY533L4DQQX2CSW5IMJ6CB2PQ, seg_pos={0 0}, upload=494622]: Done uploading piece. err: <nil> Dec 29 03:37:45.199163818 (564h8m45.070246853s) [noise=true , address=145.79.227.7:4297 , piece_id=NNJ2KDDLNZHT4SGOUXBWBYD5OGW6MTGCAWZ63D5LH7BGM6KOJGDA, seg_pos={0 0}, upload=494622]: Done uploading piece. err: <nil> Dec 29 03:37:45.448727893 (564h8m45.319810928s) [noise=true , address=45.91.116.11:4139 , piece_id=S3MEER2QCVCORYQNYNY2S22TC2H5HHKZW7VQJ54VL754UZTD42JQ, seg_pos={0 0}, upload=494622]: Done uploading piece. err: <nil> Dec 29 03:37:45.459119140 (564h8m45.330202175s) [noise=true , address=77.90.31.13:4649 , piece_id=S7T55BAWP53YJVHPW2GPLUM2VB3NBHZ2YD4UWBKKWCFHDAXAW3KQ, seg_pos={0 0}, upload=494622]: Done uploading piece. err: <nil> Dec 29 03:37:45.471609714 (564h8m45.342692759s) [noise=true , address=45.91.116.8:4050 , piece_id=NYWKOTPLBZQIHGSQFMTT2QE6OQX7H5OKFVWJK3I65CWGGDSDDBMA, seg_pos={0 0}, upload=494622]: Done uploading piece. err: <nil> Dec 29 03:37:45.489098314 (564h8m45.36018136s) [noise=true , address=77.90.31.21:4899 , piece_id=4MZKL4KVJF4DYGVPYGVOUHD6TYIDCRER33YPW6RG7KT6B6IXQEOA, seg_pos={0 0}, upload=494622]: Done uploading piece. err: <nil> Dec 29 03:37:45.489726060 (564h8m45.360809105s) [noise=true , address=77.90.31.11:4574 , piece_id=3TP2ME5NTXVK45B34W5U4DESLFJLOJEPRQOJLEEPN2PIMYS2IDWQ, seg_pos={0 0}, upload=494622]: Done uploading piece. err: <nil> Dec 29 03:37:45.779669865 (564h8m45.65075291s) [noise=true , address=145.79.227.9:4388 , piece_id=DQU5TSAWTEEIBT272I76ZMGXD6FJT2QMXQISRDMWG5NOPORLBLLA, seg_pos={0 0}, upload=494622]: Done uploading piece. err: <nil> Dec 29 03:37:45.817564973 (564h8m45.688648008s) [noise=true , address=77.90.31.18:4834 , piece_id=FQS6BFFJTORJPKJCINDZKXTY6GFRQTR3FZPWAEG7F2DNROUEPGJA, seg_pos={0 0}, upload=494622]: Done uploading piece. err: <nil> Dec 29 03:37:46.001090802 (564h8m45.872173857s) [noise=true , address=193.149.164.143:4695 , piece_id=YF4EWRMS3GWEYPSA67GLG6EA5H7GZ55GTA3ISIEJ7RFEG2NDTLEQ, seg_pos={0 0}, upload=494622]: Done uploading piece. err: <nil> Dec 29 03:37:46.129753764 (564h8m46.000836819s) [noise=true , address=145.79.227.33:4886 , piece_id=DB64UXSDC47SRTQX7PJGE66RCL7XFW3TGMCU3JOWVEJVGVHLYGWQ, seg_pos={0 0}, upload=494622]: Done uploading piece. err: <nil> Dec 29 03:37:46.163280064 (564h8m46.034363109s) [noise=true , address=145.79.227.34:4909 , piece_id=UCGJPN357USFHIQ7II5GLRKNDE4H4HUVDQ7CB5R27C3VGRP2R3IA, seg_pos={0 0}, upload=494622]: Done uploading piece. err: <nil> Dec 29 03:37:46.167742170 (564h8m46.038825205s) [noise=true , address=145.79.227.30:4790 , piece_id=2TSIHAVWOZPOVKSTG2ZXBBEFEMMMRLMOWHM47TDANARFPDJM3NPQ, seg_pos={0 0}, upload=494622]: Done uploading piece. err: <nil> Dec 29 03:37:46.167764403 (564h8m46.038847448s) [seg_pos={0 0}, upload=494622]: Segment reached optimal threshold of 14 pieces. Dec 29 03:37:46.167996534 (564h8m46.039079569s) [noise=true , address=45.91.116.10:4096 , piece_id=VIQK6EV67DONFZY77BOTWHY7POJEOE7FDHB6TKQMHVSHJJM35V7Q, seg_pos={0 0}, upload=494622]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12MZX1GwZGrEi2q44dPiYrwMoum5A2caZtXTS3qiLNcoyHhon7p): protocol: expected piece hash; context canceled; EOF Dec 29 03:37:46.168005812 (564h8m46.039088857s) [noise=true , address=45.91.116.10:4096 , piece_id=VIQK6EV67DONFZY77BOTWHY7POJEOE7FDHB6TKQMHVSHJJM35V7Q, seg_pos={0 0}, upload=494622]: Long tail cancellation due to optimal threshold Dec 29 03:37:46.168120160 (564h8m46.039203195s) [noise=true , address=64.49.13.26:4950 , piece_id=HQ6BBIFQAD5RRN7FRWGDFHAVVMSDUWJUGTX7LBTA2TQJ3PNWKUHA, seg_pos={0 0}, upload=494622]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14Tv25fLSqTnZQiDzrYqKCSbdt6ep3upgqD2Se8QSRuQFynkSN): protocol: expected piece hash; context canceled; EOF Dec 29 03:37:46.168136090 (564h8m46.039219125s) [noise=true , address=64.49.13.26:4950 , piece_id=HQ6BBIFQAD5RRN7FRWGDFHAVVMSDUWJUGTX7LBTA2TQJ3PNWKUHA, seg_pos={0 0}, upload=494622]: Long tail cancellation due to optimal threshold Dec 29 03:37:46.168274995 (564h8m46.03935804s) [noise=true , address=77.90.31.12:4600 , piece_id=N4KXD5SYPYOKBXQRSL3UKNDDV7XB4KOKRYRYKGMMAHMAO6VVOEHQ, seg_pos={0 0}, upload=494622]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PKT2EcU6jAe6oUUdiTcwxGQsFa36uEmown77SemqQMM5Xw5nc): protocol: expected piece hash; context canceled; EOF Dec 29 03:37:46.168283491 (564h8m46.039366536s) [noise=true , address=77.90.31.12:4600 , piece_id=N4KXD5SYPYOKBXQRSL3UKNDDV7XB4KOKRYRYKGMMAHMAO6VVOEHQ, seg_pos={0 0}, upload=494622]: Long tail cancellation due to optimal threshold Dec 29 03:37:46.168394032 (564h8m46.039477067s) [noise=true , address=145.79.227.40:4068 , piece_id=4QOJWRVEE3ERM2AHP764XSTE7ZYHUUJ2ENBVXRS5UDIGKZOK7YKA, seg_pos={0 0}, upload=494622]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Lh9dYpHCxbs7gpZ4Xt7os5Gm5Nco6jrNW1V1FHddVUEMkFfPY): protocol: expected piece hash; context canceled; EOF Dec 29 03:37:46.168403810 (564h8m46.039486845s) [noise=true , address=145.79.227.40:4068 , piece_id=4QOJWRVEE3ERM2AHP764XSTE7ZYHUUJ2ENBVXRS5UDIGKZOK7YKA, seg_pos={0 0}, upload=494622]: Long tail cancellation due to optimal threshold Dec 29 03:37:46.168432725 (564h8m46.03951577s) [seg_pos={0 0}, upload=494622]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 03:37:46.168576960 (564h8m46.039659995s) [upload=494622]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 03:37:46.168584033 (564h8m46.039667068s) [upload=494622]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 03:37:46.466683658 (564h8m46.337766723s) [upload=494622]: Done uploading segments... Dec 29 03:37:46.466712052 (564h8m46.337795087s) [upload=494622]: Upload finished. err: <nil> Duration: 54.308019497s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [..............................XX................X......XXX..] duration: 3.67606323s trace: https://jaeger.dev.storj.io/trace/79e92355ce966062 FAIL (2025-12-29 03:42:10 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.676087726s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 29 out of 60 is failed (threshold=3) [.XX.....XX.X.....X.X..XXXX..XXX.XX...X.X.....XXX...XXXXXXX.X] duration: 30.003208574s trace: https://jaeger.dev.storj.io/trace/5a528fd931efbef6 FAIL (2025-12-29 03:42:02 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== egments... Dec 29 03:41:32.921622196 (564h12m32.792705231s) [upload=494672]: Got next segment. Inline: false Dec 29 03:41:32.921657904 (564h12m32.792740939s) [upload=494672]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 03:41:32.921662473 (564h12m32.792745508s) [upload=494672]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 03:41:33.166169720 (564h12m33.037252765s) [seg_pos={0 0}, upload=494672]: Begin upload segment... Dec 29 03:41:33.166234723 (564h12m33.037317768s) [seg_pos={0 0}, upload=494672]: Done begin upload segment. Dec 29 03:41:33.166305107 (564h12m33.037388142s) [noise=true , address=77.90.31.18:4808 , piece_id=K2BDFMQJIAPG3LBLMM5PQ2ZZSHFPKYPLEBE7PGVEPY53C5CZWELA, seg_pos={0 0}, upload=494672]: Uploading piece... Dec 29 03:41:33.166342449 (564h12m33.037425484s) [noise=true , address=45.91.116.13:4154 , piece_id=OSIWM7CRMRYLWY2WYR3TUKUVGWCUNPUR2MY6FRKHIYDYDJ3X7OWA, seg_pos={0 0}, upload=494672]: Uploading piece... Dec 29 03:41:33.166968630 (564h12m33.038051665s) [noise=true , address=77.90.31.14:4664 , piece_id=Q6ESBH2NCOONBRRBGOMY3IOPS4FORFDX7INR43AQJ7MZYY4OLFXA, seg_pos={0 0}, upload=494672]: Uploading piece... Dec 29 03:41:33.167396986 (564h12m33.038480021s) [noise=true , address=145.79.227.45:4230 , piece_id=G64OOU3ZNMJX5AJH6NMYZV4GBKMYCSJJJJSEPAKVH4MPSOMC7EJQ, seg_pos={0 0}, upload=494672]: Uploading piece... Dec 29 03:41:33.167818198 (564h12m33.038901243s) [noise=true , address=45.91.116.8:4046 , piece_id=4EZYQTP35ZOSZSULEFJH34WUHZZNQGGEHSCAUPQAKAHAL4GI4K2A, seg_pos={0 0}, upload=494672]: Uploading piece... Dec 29 03:41:33.168216136 (564h12m33.039299171s) [noise=true , address=145.79.227.36:4519 , piece_id=FIIBFWZYLSXR5Y5BWJZTYOTKIFHHWGGBTQHJBV5KELHEMRL4RP6Q, seg_pos={0 0}, upload=494672]: Uploading piece... Dec 29 03:41:33.168622279 (564h12m33.039705314s) [noise=true , address=145.79.227.12:4765 , piece_id=WAJ2ACT4RPIG6RGNSKIGUON5VIMNGOOD5TMBGG5XM6SCQVGYZDZQ, seg_pos={0 0}, upload=494672]: Uploading piece... Dec 29 03:41:33.169025467 (564h12m33.040108503s) [noise=true , address=45.91.116.5:4299 , piece_id=PHYVUVB3YUV4K3KYIV4VRBWOT6CS3XQZBGOYC7Q6BXCBSQ6PVI5A, seg_pos={0 0}, upload=494672]: Uploading piece... Dec 29 03:41:33.170458005 (564h12m33.04154105s) [noise=true , address=145.79.227.52:4447 , piece_id=7OOJPQOFJCESUG7I4QFXNZNLQTEMD3NRZMZ7Z3BTBZYBJNJTT5UQ, seg_pos={0 0}, upload=494672]: Uploading piece... Dec 29 03:41:33.171982467 (564h12m33.043065512s) [noise=true , address=145.79.227.32:4834 , piece_id=DIPRKSJURP6YEZSAOB7FUSDFOCHSUTS56MSJJ7JGKCHDU6NHEUIA, seg_pos={0 0}, upload=494672]: Uploading piece... Dec 29 03:41:33.173405747 (564h12m33.044488782s) [noise=true , address=145.79.227.51:4409 , piece_id=YJ4XZ2EBXFJF6QOF23EJFR2Z6YYUC7PX2JSKEUYRPYMVJKRKXXVQ, seg_pos={0 0}, upload=494672]: Uploading piece... Dec 29 03:41:33.175296007 (564h12m33.046379052s) [noise=true , address=45.91.116.12:4226 , piece_id=23UOEUZZCQKZ7GFGXKVXLEV6QUPD4CRS4BEP2BUUIF6E2W3YVBEQ, seg_pos={0 0}, upload=494672]: Uploading piece... Dec 29 03:41:33.176959574 (564h12m33.04804261s) [noise=true , address=145.79.227.47:4265 , piece_id=N3V77YFNNIQV42EZR76PFKB35DLJJLCDBZTLHVAVJLDUH4XLWZDQ, seg_pos={0 0}, upload=494672]: Uploading piece... Dec 29 03:41:33.178462186 (564h12m33.049545221s) [noise=true , address=77.90.31.19:4842 , piece_id=LDOXHKOAUNU2GR22AAAJRRMXLG6PXMFLR5WKAT7R7PXEEJ5FL2AA, seg_pos={0 0}, upload=494672]: Uploading piece... Dec 29 03:41:33.179053391 (564h12m33.050136426s) [noise=true , address=145.79.227.2:4115 , piece_id=DQZVUFVNXILHRRMQTWOUEYDIBE3CWFRLSDO2ZME3JM3BSGKJ6ZJA, seg_pos={0 0}, upload=494672]: Uploading piece... Dec 29 03:41:33.180698203 (564h12m33.051781238s) [noise=true , address=45.91.116.9:4263 , piece_id=DTD3GSZUC7QO3TR3BXSYOP5XXXSAIZZE7KMCA7F6Z6OS7ZHZD2QQ, seg_pos={0 0}, upload=494672]: Uploading piece... Dec 29 03:41:34.873875136 (564h12m34.744958171s) [upload=494672]: Next returned nil segment Dec 29 03:41:34.873898831 (564h12m34.744981876s) [upload=494672]: Waiting for error group managing segments... Dec 29 03:41:35.151180225 (564h12m35.02226326s) [noise=true , address=145.79.227.32:4834 , piece_id=DIPRKSJURP6YEZSAOB7FUSDFOCHSUTS56MSJJ7JGKCHDU6NHEUIA, seg_pos={0 0}, upload=494672]: Done uploading piece. err: <nil> Dec 29 03:41:35.153860939 (564h12m35.024943984s) [noise=true , address=145.79.227.52:4447 , piece_id=7OOJPQOFJCESUG7I4QFXNZNLQTEMD3NRZMZ7Z3BTBZYBJNJTT5UQ, seg_pos={0 0}, upload=494672]: Done uploading piece. err: <nil> Dec 29 03:41:35.159842067 (564h12m35.030925102s) [noise=true , address=145.79.227.36:4519 , piece_id=FIIBFWZYLSXR5Y5BWJZTYOTKIFHHWGGBTQHJBV5KELHEMRL4RP6Q, seg_pos={0 0}, upload=494672]: Done uploading piece. err: <nil> Dec 29 03:41:35.164950335 (564h12m35.03603337s) [noise=true , address=145.79.227.51:4409 , piece_id=YJ4XZ2EBXFJF6QOF23EJFR2Z6YYUC7PX2JSKEUYRPYMVJKRKXXVQ, seg_pos={0 0}, upload=494672]: Done uploading piece. err: <nil> Dec 29 03:41:35.168324910 (564h12m35.039407956s) [noise=true , address=145.79.227.2:4115 , piece_id=DQZVUFVNXILHRRMQTWOUEYDIBE3CWFRLSDO2ZME3JM3BSGKJ6ZJA, seg_pos={0 0}, upload=494672]: Done uploading piece. err: <nil> Dec 29 03:41:35.175094981 (564h12m35.046178016s) [noise=true , address=145.79.227.12:4765 , piece_id=WAJ2ACT4RPIG6RGNSKIGUON5VIMNGOOD5TMBGG5XM6SCQVGYZDZQ, seg_pos={0 0}, upload=494672]: Done uploading piece. err: <nil> Dec 29 03:41:35.181937752 (564h12m35.053020797s) [noise=true , address=145.79.227.45:4230 , piece_id=G64OOU3ZNMJX5AJH6NMYZV4GBKMYCSJJJJSEPAKVH4MPSOMC7EJQ, seg_pos={0 0}, upload=494672]: Done uploading piece. err: <nil> Dec 29 03:41:35.196310058 (564h12m35.067393093s) [noise=true , address=145.79.227.47:4265 , piece_id=N3V77YFNNIQV42EZR76PFKB35DLJJLCDBZTLHVAVJLDUH4XLWZDQ, seg_pos={0 0}, upload=494672]: Done uploading piece. err: <nil> Dec 29 03:41:35.450223316 (564h12m35.321306361s) [noise=true , address=45.91.116.13:4154 , piece_id=OSIWM7CRMRYLWY2WYR3TUKUVGWCUNPUR2MY6FRKHIYDYDJ3X7OWA, seg_pos={0 0}, upload=494672]: Done uploading piece. err: <nil> Dec 29 03:41:35.553040835 (564h12m35.42412388s) [noise=true , address=77.90.31.18:4808 , piece_id=K2BDFMQJIAPG3LBLMM5PQ2ZZSHFPKYPLEBE7PGVEPY53C5CZWELA, seg_pos={0 0}, upload=494672]: Done uploading piece. err: <nil> Dec 29 03:41:35.565436559 (564h12m35.436519594s) [noise=true , address=45.91.116.8:4046 , piece_id=4EZYQTP35ZOSZSULEFJH34WUHZZNQGGEHSCAUPQAKAHAL4GI4K2A, seg_pos={0 0}, upload=494672]: Done uploading piece. err: <nil> Dec 29 03:41:35.646843710 (564h12m35.517926745s) [noise=true , address=45.91.116.5:4299 , piece_id=PHYVUVB3YUV4K3KYIV4VRBWOT6CS3XQZBGOYC7Q6BXCBSQ6PVI5A, seg_pos={0 0}, upload=494672]: Done uploading piece. err: <nil> Dec 29 03:41:35.709659500 (564h12m35.580742535s) [noise=true , address=45.91.116.9:4263 , piece_id=DTD3GSZUC7QO3TR3BXSYOP5XXXSAIZZE7KMCA7F6Z6OS7ZHZD2QQ, seg_pos={0 0}, upload=494672]: Done uploading piece. err: <nil> Dec 29 03:41:35.709681161 (564h12m35.580764206s) [seg_pos={0 0}, upload=494672]: Segment reached optimal threshold of 13 pieces. Dec 29 03:41:35.709907743 (564h12m35.580990778s) [noise=true , address=45.91.116.12:4226 , piece_id=23UOEUZZCQKZ7GFGXKVXLEV6QUPD4CRS4BEP2BUUIF6E2W3YVBEQ, seg_pos={0 0}, upload=494672]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: write tcp 188.166.225.170:55042->45.91.116.12:4226: use of closed network connection; write tcp 188.166.225.170:55042->45.91.116.12:4226: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55042->45.91.116.12:4226: use of closed network connection Dec 29 03:41:35.709915758 (564h12m35.580998793s) [noise=true , address=45.91.116.12:4226 , piece_id=23UOEUZZCQKZ7GFGXKVXLEV6QUPD4CRS4BEP2BUUIF6E2W3YVBEQ, seg_pos={0 0}, upload=494672]: Long tail cancellation due to optimal threshold Dec 29 03:41:35.709951235 (564h12m35.58103427s) [noise=true , address=77.90.31.14:4664 , piece_id=Q6ESBH2NCOONBRRBGOMY3IOPS4FORFDX7INR43AQJ7MZYY4OLFXA, seg_pos={0 0}, upload=494672]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9): protocol: expected piece hash; context canceled; EOF Dec 29 03:41:35.709960312 (564h12m35.581043347s) [noise=true , address=77.90.31.14:4664 , piece_id=Q6ESBH2NCOONBRRBGOMY3IOPS4FORFDX7INR43AQJ7MZYY4OLFXA, seg_pos={0 0}, upload=494672]: Long tail cancellation due to optimal threshold Dec 29 03:41:35.710110890 (564h12m35.581193925s) [noise=true , address=77.90.31.19:4842 , piece_id=LDOXHKOAUNU2GR22AAAJRRMXLG6PXMFLR5WKAT7R7PXEEJ5FL2AA, seg_pos={0 0}, upload=494672]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: write tcp 188.166.225.170:54970->77.90.31.19:4842: use of closed network connection; write tcp 188.166.225.170:54970->77.90.31.19:4842: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54970->77.90.31.19:4842: use of closed network connection Dec 29 03:41:35.710117342 (564h12m35.581200377s) [noise=true , address=77.90.31.19:4842 , piece_id=LDOXHKOAUNU2GR22AAAJRRMXLG6PXMFLR5WKAT7R7PXEEJ5FL2AA, seg_pos={0 0}, upload=494672]: Long tail cancellation due to optimal threshold Dec 29 03:41:35.710140416 (564h12m35.581223461s) [seg_pos={0 0}, upload=494672]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 03:41:35.710215719 (564h12m35.581298754s) [upload=494672]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 03:41:35.710220458 (564h12m35.581303503s) [upload=494672]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 03:41:36.006135012 (564h12m35.877218057s) [upload=494672]: Done uploading segments... Dec 29 03:41:36.006159699 (564h12m35.877242734s) [upload=494672]: Upload finished. err: <nil> Duration: 30.003233592s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [...............................XX................X......XXX.] duration: 3.583572198s trace: https://jaeger.dev.storj.io/trace/79e92355ce966062 FAIL (2025-12-29 03:41:50 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.58360433s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [....................................X...................X...] duration: 31.97879171s trace: https://jaeger.dev.storj.io/trace/2408171d336afafd FAIL (2025-12-29 03:41:20 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12Tz3HEHACmHU9Gp2W2vHVuF5PXNmdsa5N8nYhwt52HCTjggqRD, Piece ID: S3MEER2QCVCORYQNYNY2S22TC2H5HHKZW7VQJ54VL754UZTD42JQ) context deadline exceeded; piecestore: (Node ID: 1ve7KKYDbNo8Ma7wGVBH5x9z7zbPtPAPztmuQ9r6ijPpVgh5NX, Piece ID: S7T55BAWP53YJVHPW2GPLUM2VB3NBHZ2YD4UWBKKWCFHDAXAW3KQ) context deadline exceeded; piecestore: (Node ID: 1WVHPacX35JJ8PFzCs3UtGwsR4yVHXKmiqfGedK9KfXbHNJtmu, Piece ID: NYWKOTPLBZQIHGSQFMTT2QE6OQX7H5OKFVWJK3I65CWGGDSDDBMA) context deadline exceeded ======LOGS====== upload=494622]: Uploading piece... Dec 29 03:37:43.315384541 (564h8m43.186467596s) [noise=true , address=145.79.227.11:4762 , piece_id=2GQ5ASSBMUC52NCYRLBF3EOKYEA3G6DTIA6K56DE3OUQ4F5OLX2Q, seg_pos={0 0}, upload=494622]: Uploading piece... Dec 29 03:37:43.315899872 (564h8m43.186982907s) [noise=true , address=77.90.31.13:4649 , piece_id=S7T55BAWP53YJVHPW2GPLUM2VB3NBHZ2YD4UWBKKWCFHDAXAW3KQ, seg_pos={0 0}, upload=494622]: Uploading piece... Dec 29 03:37:43.316395165 (564h8m43.1874782s) [noise=true , address=77.90.31.18:4834 , piece_id=FQS6BFFJTORJPKJCINDZKXTY6GFRQTR3FZPWAEG7F2DNROUEPGJA, seg_pos={0 0}, upload=494622]: Uploading piece... Dec 29 03:37:43.316885961 (564h8m43.187969006s) [noise=true , address=193.149.164.143:4695 , piece_id=YF4EWRMS3GWEYPSA67GLG6EA5H7GZ55GTA3ISIEJ7RFEG2NDTLEQ, seg_pos={0 0}, upload=494622]: Uploading piece... Dec 29 03:37:43.317355795 (564h8m43.18843883s) [noise=true , address=145.79.227.7:4297 , piece_id=NNJ2KDDLNZHT4SGOUXBWBYD5OGW6MTGCAWZ63D5LH7BGM6KOJGDA, seg_pos={0 0}, upload=494622]: Uploading piece... Dec 29 03:37:43.317364631 (564h8m43.188447666s) [noise=true , address=145.79.227.33:4886 , piece_id=DB64UXSDC47SRTQX7PJGE66RCL7XFW3TGMCU3JOWVEJVGVHLYGWQ, seg_pos={0 0}, upload=494622]: Uploading piece... Dec 29 03:37:43.317689451 (564h8m43.188772486s) [noise=true , address=64.49.13.26:4950 , piece_id=HQ6BBIFQAD5RRN7FRWGDFHAVVMSDUWJUGTX7LBTA2TQJ3PNWKUHA, seg_pos={0 0}, upload=494622]: Uploading piece... Dec 29 03:37:43.317877448 (564h8m43.188960493s) [noise=true , address=45.91.116.10:4096 , piece_id=VIQK6EV67DONFZY77BOTWHY7POJEOE7FDHB6TKQMHVSHJJM35V7Q, seg_pos={0 0}, upload=494622]: Uploading piece... Dec 29 03:37:43.318315813 (564h8m43.189398858s) [noise=true , address=145.79.227.9:4388 , piece_id=DQU5TSAWTEEIBT272I76ZMGXD6FJT2QMXQISRDMWG5NOPORLBLLA, seg_pos={0 0}, upload=494622]: Uploading piece... Dec 29 03:37:43.318722317 (564h8m43.189805352s) [noise=true , address=77.90.31.12:4600 , piece_id=N4KXD5SYPYOKBXQRSL3UKNDDV7XB4KOKRYRYKGMMAHMAO6VVOEHQ, seg_pos={0 0}, upload=494622]: Uploading piece... Dec 29 03:37:43.319195248 (564h8m43.190278293s) [noise=true , address=45.91.116.8:4050 , piece_id=NYWKOTPLBZQIHGSQFMTT2QE6OQX7H5OKFVWJK3I65CWGGDSDDBMA, seg_pos={0 0}, upload=494622]: Uploading piece... Dec 29 03:37:43.319666174 (564h8m43.190749219s) [noise=true , address=77.90.31.21:4899 , piece_id=4MZKL4KVJF4DYGVPYGVOUHD6TYIDCRER33YPW6RG7KT6B6IXQEOA, seg_pos={0 0}, upload=494622]: Uploading piece... Dec 29 03:37:43.321557184 (564h8m43.192640229s) [noise=true , address=145.79.227.34:4909 , piece_id=UCGJPN357USFHIQ7II5GLRKNDE4H4HUVDQ7CB5R27C3VGRP2R3IA, seg_pos={0 0}, upload=494622]: Uploading piece... Dec 29 03:37:43.322231088 (564h8m43.193314133s) [noise=true , address=45.91.116.11:4139 , piece_id=S3MEER2QCVCORYQNYNY2S22TC2H5HHKZW7VQJ54VL754UZTD42JQ, seg_pos={0 0}, upload=494622]: Uploading piece... Dec 29 03:37:43.322851720 (564h8m43.193934755s) [noise=true , address=145.79.227.51:4444 , piece_id=EWEXXYYWF43J7FG6NPSJ2PICHTRPY533L4DQQX2CSW5IMJ6CB2PQ, seg_pos={0 0}, upload=494622]: Uploading piece... Dec 29 03:37:43.323435972 (564h8m43.194519007s) [noise=true , address=145.79.227.40:4068 , piece_id=4QOJWRVEE3ERM2AHP764XSTE7ZYHUUJ2ENBVXRS5UDIGKZOK7YKA, seg_pos={0 0}, upload=494622]: Uploading piece... Dec 29 03:37:43.323901118 (564h8m43.194984153s) [noise=true , address=145.79.227.30:4790 , piece_id=2TSIHAVWOZPOVKSTG2ZXBBEFEMMMRLMOWHM47TDANARFPDJM3NPQ, seg_pos={0 0}, upload=494622]: Uploading piece... Dec 29 03:37:44.620573279 (564h8m44.491656314s) [upload=494622]: Next returned nil segment Dec 29 03:37:44.620593617 (564h8m44.491676652s) [upload=494622]: Waiting for error group managing segments... Dec 29 03:37:45.178264102 (564h8m45.049347137s) [noise=true , address=145.79.227.11:4762 , piece_id=2GQ5ASSBMUC52NCYRLBF3EOKYEA3G6DTIA6K56DE3OUQ4F5OLX2Q, seg_pos={0 0}, upload=494622]: Done uploading piece. err: <nil> Dec 29 03:37:45.190189951 (564h8m45.061272986s) [noise=true , address=145.79.227.51:4444 , piece_id=EWEXXYYWF43J7FG6NPSJ2PICHTRPY533L4DQQX2CSW5IMJ6CB2PQ, seg_pos={0 0}, upload=494622]: Done uploading piece. err: <nil> Dec 29 03:37:45.199163818 (564h8m45.070246853s) [noise=true , address=145.79.227.7:4297 , piece_id=NNJ2KDDLNZHT4SGOUXBWBYD5OGW6MTGCAWZ63D5LH7BGM6KOJGDA, seg_pos={0 0}, upload=494622]: Done uploading piece. err: <nil> Dec 29 03:37:45.448727893 (564h8m45.319810928s) [noise=true , address=45.91.116.11:4139 , piece_id=S3MEER2QCVCORYQNYNY2S22TC2H5HHKZW7VQJ54VL754UZTD42JQ, seg_pos={0 0}, upload=494622]: Done uploading piece. err: <nil> Dec 29 03:37:45.459119140 (564h8m45.330202175s) [noise=true , address=77.90.31.13:4649 , piece_id=S7T55BAWP53YJVHPW2GPLUM2VB3NBHZ2YD4UWBKKWCFHDAXAW3KQ, seg_pos={0 0}, upload=494622]: Done uploading piece. err: <nil> Dec 29 03:37:45.471609714 (564h8m45.342692759s) [noise=true , address=45.91.116.8:4050 , piece_id=NYWKOTPLBZQIHGSQFMTT2QE6OQX7H5OKFVWJK3I65CWGGDSDDBMA, seg_pos={0 0}, upload=494622]: Done uploading piece. err: <nil> Dec 29 03:37:45.489098314 (564h8m45.36018136s) [noise=true , address=77.90.31.21:4899 , piece_id=4MZKL4KVJF4DYGVPYGVOUHD6TYIDCRER33YPW6RG7KT6B6IXQEOA, seg_pos={0 0}, upload=494622]: Done uploading piece. err: <nil> Dec 29 03:37:45.489726060 (564h8m45.360809105s) [noise=true , address=77.90.31.11:4574 , piece_id=3TP2ME5NTXVK45B34W5U4DESLFJLOJEPRQOJLEEPN2PIMYS2IDWQ, seg_pos={0 0}, upload=494622]: Done uploading piece. err: <nil> Dec 29 03:37:45.779669865 (564h8m45.65075291s) [noise=true , address=145.79.227.9:4388 , piece_id=DQU5TSAWTEEIBT272I76ZMGXD6FJT2QMXQISRDMWG5NOPORLBLLA, seg_pos={0 0}, upload=494622]: Done uploading piece. err: <nil> Dec 29 03:37:45.817564973 (564h8m45.688648008s) [noise=true , address=77.90.31.18:4834 , piece_id=FQS6BFFJTORJPKJCINDZKXTY6GFRQTR3FZPWAEG7F2DNROUEPGJA, seg_pos={0 0}, upload=494622]: Done uploading piece. err: <nil> Dec 29 03:37:46.001090802 (564h8m45.872173857s) [noise=true , address=193.149.164.143:4695 , piece_id=YF4EWRMS3GWEYPSA67GLG6EA5H7GZ55GTA3ISIEJ7RFEG2NDTLEQ, seg_pos={0 0}, upload=494622]: Done uploading piece. err: <nil> Dec 29 03:37:46.129753764 (564h8m46.000836819s) [noise=true , address=145.79.227.33:4886 , piece_id=DB64UXSDC47SRTQX7PJGE66RCL7XFW3TGMCU3JOWVEJVGVHLYGWQ, seg_pos={0 0}, upload=494622]: Done uploading piece. err: <nil> Dec 29 03:37:46.163280064 (564h8m46.034363109s) [noise=true , address=145.79.227.34:4909 , piece_id=UCGJPN357USFHIQ7II5GLRKNDE4H4HUVDQ7CB5R27C3VGRP2R3IA, seg_pos={0 0}, upload=494622]: Done uploading piece. err: <nil> Dec 29 03:37:46.167742170 (564h8m46.038825205s) [noise=true , address=145.79.227.30:4790 , piece_id=2TSIHAVWOZPOVKSTG2ZXBBEFEMMMRLMOWHM47TDANARFPDJM3NPQ, seg_pos={0 0}, upload=494622]: Done uploading piece. err: <nil> Dec 29 03:37:46.167764403 (564h8m46.038847448s) [seg_pos={0 0}, upload=494622]: Segment reached optimal threshold of 14 pieces. Dec 29 03:37:46.167996534 (564h8m46.039079569s) [noise=true , address=45.91.116.10:4096 , piece_id=VIQK6EV67DONFZY77BOTWHY7POJEOE7FDHB6TKQMHVSHJJM35V7Q, seg_pos={0 0}, upload=494622]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12MZX1GwZGrEi2q44dPiYrwMoum5A2caZtXTS3qiLNcoyHhon7p): protocol: expected piece hash; context canceled; EOF Dec 29 03:37:46.168005812 (564h8m46.039088857s) [noise=true , address=45.91.116.10:4096 , piece_id=VIQK6EV67DONFZY77BOTWHY7POJEOE7FDHB6TKQMHVSHJJM35V7Q, seg_pos={0 0}, upload=494622]: Long tail cancellation due to optimal threshold Dec 29 03:37:46.168120160 (564h8m46.039203195s) [noise=true , address=64.49.13.26:4950 , piece_id=HQ6BBIFQAD5RRN7FRWGDFHAVVMSDUWJUGTX7LBTA2TQJ3PNWKUHA, seg_pos={0 0}, upload=494622]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14Tv25fLSqTnZQiDzrYqKCSbdt6ep3upgqD2Se8QSRuQFynkSN): protocol: expected piece hash; context canceled; EOF Dec 29 03:37:46.168136090 (564h8m46.039219125s) [noise=true , address=64.49.13.26:4950 , piece_id=HQ6BBIFQAD5RRN7FRWGDFHAVVMSDUWJUGTX7LBTA2TQJ3PNWKUHA, seg_pos={0 0}, upload=494622]: Long tail cancellation due to optimal threshold Dec 29 03:37:46.168274995 (564h8m46.03935804s) [noise=true , address=77.90.31.12:4600 , piece_id=N4KXD5SYPYOKBXQRSL3UKNDDV7XB4KOKRYRYKGMMAHMAO6VVOEHQ, seg_pos={0 0}, upload=494622]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PKT2EcU6jAe6oUUdiTcwxGQsFa36uEmown77SemqQMM5Xw5nc): protocol: expected piece hash; context canceled; EOF Dec 29 03:37:46.168283491 (564h8m46.039366536s) [noise=true , address=77.90.31.12:4600 , piece_id=N4KXD5SYPYOKBXQRSL3UKNDDV7XB4KOKRYRYKGMMAHMAO6VVOEHQ, seg_pos={0 0}, upload=494622]: Long tail cancellation due to optimal threshold Dec 29 03:37:46.168394032 (564h8m46.039477067s) [noise=true , address=145.79.227.40:4068 , piece_id=4QOJWRVEE3ERM2AHP764XSTE7ZYHUUJ2ENBVXRS5UDIGKZOK7YKA, seg_pos={0 0}, upload=494622]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Lh9dYpHCxbs7gpZ4Xt7os5Gm5Nco6jrNW1V1FHddVUEMkFfPY): protocol: expected piece hash; context canceled; EOF Dec 29 03:37:46.168403810 (564h8m46.039486845s) [noise=true , address=145.79.227.40:4068 , piece_id=4QOJWRVEE3ERM2AHP764XSTE7ZYHUUJ2ENBVXRS5UDIGKZOK7YKA, seg_pos={0 0}, upload=494622]: Long tail cancellation due to optimal threshold Dec 29 03:37:46.168432725 (564h8m46.03951577s) [seg_pos={0 0}, upload=494622]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 03:37:46.168576960 (564h8m46.039659995s) [upload=494622]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 03:37:46.168584033 (564h8m46.039667068s) [upload=494622]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 03:37:46.466683658 (564h8m46.337766723s) [upload=494622]: Done uploading segments... Dec 29 03:37:46.466712052 (564h8m46.337795087s) [upload=494622]: Upload finished. err: <nil> Duration: 31.978819413s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [................................XX................X......XXX] duration: 5.002788055s trace: https://jaeger.dev.storj.io/trace/79e92355ce966062 FAIL (2025-12-29 03:41:16 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.002812572s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 28 out of 60 is failed (threshold=3) [..XX.....XX.X.....X.X..XXXX..XXX.XX...X.X.....XXX...XXXXXXX.] duration: 14.315144171s trace: https://jaeger.dev.storj.io/trace/5d868317ce787ff0 FAIL (2025-12-29 03:40:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, Piece ID: 3R6ASNVD23AVWVK4AHBMQJCH3TV2YSHM4SDZHK45AIO7ON2BM7BA) context deadline exceeded; piecestore: (Node ID: 1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, Piece ID: DM32E2LNQNXEPSIRVFVW3RHNWXCH725MIWCR2VYDKE6JA2FDR6VA) context deadline exceeded; piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: I3ERFWTC5OWOKWGTU6ME2QJQFLROHHH5XUDTIGJGNP5YS5WMGQDA) context deadline exceeded ======LOGS====== 29 03:39:25.931217811 (564h10m25.802300846s) [upload=494645]: Starting upload Dec 29 03:39:25.931273377 (564h10m25.802356412s) [upload=494645]: Done starting upload Dec 29 03:39:25.931469391 (564h10m25.802552426s) [upload=494645]: Uploading segments... Dec 29 03:39:25.931711250 (564h10m25.802794296s) [upload=494645]: Got next segment. Inline: false Dec 29 03:39:25.931762458 (564h10m25.802845503s) [upload=494645]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 03:39:25.931769441 (564h10m25.802852897s) [upload=494645]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 03:39:26.174866223 (564h10m26.045949258s) [seg_pos={0 0}, upload=494645]: Begin upload segment... Dec 29 03:39:26.174911159 (564h10m26.045994204s) [seg_pos={0 0}, upload=494645]: Done begin upload segment. Dec 29 03:39:26.174922570 (564h10m26.046005615s) [noise=true , address=145.79.227.51:4411 , piece_id=LDPNKCXBIO5UBDEXUBAOOLY6342VBXTTNQB5P3HQ6JMI5HRQK7WQ, seg_pos={0 0}, upload=494645]: Uploading piece... Dec 29 03:39:26.174937790 (564h10m26.046020835s) [noise=true , address=145.79.227.26:4653 , piece_id=WDQQCWBA4KEEBDUVDPGI6SKV2IIPZE57NYYB2DFH2BG4M3P3B2HQ, seg_pos={0 0}, upload=494645]: Uploading piece... Dec 29 03:39:26.175624657 (564h10m26.046707692s) [noise=true , address=45.91.116.9:4264 , piece_id=X35QO5ECUSQQ6J5E3XOZCGGSU3HNOIBGPTOOQO5QIH6MY2V54KRA, seg_pos={0 0}, upload=494645]: Uploading piece... Dec 29 03:39:26.175941531 (564h10m26.047024566s) [noise=true , address=77.90.31.11:4554 , piece_id=DM32E2LNQNXEPSIRVFVW3RHNWXCH725MIWCR2VYDKE6JA2FDR6VA, seg_pos={0 0}, upload=494645]: Uploading piece... Dec 29 03:39:26.176222386 (564h10m26.047305421s) [noise=true , address=145.79.227.11:4730 , piece_id=VCXWUKAUMND4EZUJKLMBTI3I7GDH62JRILOFL4IVSRXLHQ2BCCIA, seg_pos={0 0}, upload=494645]: Uploading piece... Dec 29 03:39:26.176509432 (564h10m26.047592477s) [noise=true , address=145.79.227.7:4295 , piece_id=7CZCFR2IZZ3FIXKNJFGH5XJTK7LOMR4XSXR3QRVFROFEOGYEMMOQ, seg_pos={0 0}, upload=494645]: Uploading piece... Dec 29 03:39:26.176782362 (564h10m26.047865397s) [noise=true , address=145.79.227.19:4078 , piece_id=Z32HCP5MBSQRPO7YHLOTEARDTJDQTI4FS5P64SRDOI26U6YXFRTA, seg_pos={0 0}, upload=494645]: Uploading piece... Dec 29 03:39:26.177049660 (564h10m26.048132705s) [noise=true , address=145.79.227.47:4266 , piece_id=IMUISAR23BUUOJYS65UCZM5CUBN7YLA2K5BA44ZWCG46LEHBHHZA, seg_pos={0 0}, upload=494645]: Uploading piece... Dec 29 03:39:26.177387153 (564h10m26.048470188s) [noise=true , address=145.79.227.2:4113 , piece_id=U2MVC6AWVKVR2P5LMDQBPN43SVO2BBSLCZJIXLK62VXBYMIJ4MOA, seg_pos={0 0}, upload=494645]: Uploading piece... Dec 29 03:39:26.177834585 (564h10m26.04891763s) [noise=true , address=45.91.116.8:4045 , piece_id=I3ERFWTC5OWOKWGTU6ME2QJQFLROHHH5XUDTIGJGNP5YS5WMGQDA, seg_pos={0 0}, upload=494645]: Uploading piece... Dec 29 03:39:26.178904823 (564h10m26.049987859s) [noise=true , address=45.91.116.11:4117 , piece_id=3YGZ3H4YC3QLM6NZPA3MZU7RB2VPNBY3IYVQEYGKT3HNGTPZFI6A, seg_pos={0 0}, upload=494645]: Uploading piece... Dec 29 03:39:26.179245491 (564h10m26.050328526s) [noise=true , address=145.79.227.45:4230 , piece_id=DFMDFGRYGEJV5TXE6KFYNV6IU7PVULNOQCVKCQU7AJIHZJPI35MQ, seg_pos={0 0}, upload=494645]: Uploading piece... Dec 29 03:39:26.180029956 (564h10m26.051112991s) [noise=true , address=145.79.227.50:4374 , piece_id=7A5FFDBUYX4M63UBGLZAZJIRWZ3TOZBISSG35G7IXSALNY63YT2Q, seg_pos={0 0}, upload=494645]: Uploading piece... Dec 29 03:39:26.180527964 (564h10m26.051610999s) [noise=true , address=45.91.116.14:4191 , piece_id=3R6ASNVD23AVWVK4AHBMQJCH3TV2YSHM4SDZHK45AIO7ON2BM7BA, seg_pos={0 0}, upload=494645]: Uploading piece... Dec 29 03:39:26.181227546 (564h10m26.052310591s) [noise=true , address=145.79.227.46:4601 , piece_id=XSW3FCQEJL3TE7TFP3LFVEGQA2XPW4TK5KZAJ2GDB2OHDHYTHHFQ, seg_pos={0 0}, upload=494645]: Uploading piece... Dec 29 03:39:26.182419296 (564h10m26.053502341s) [noise=true , address=77.90.31.10:4518 , piece_id=BE6Z5FDNUBQ4O2QFPQBRTYLNSENP3UVHHOVAPMC7WBEMKCCZ2QDQ, seg_pos={0 0}, upload=494645]: Uploading piece... Dec 29 03:39:27.818937762 (564h10m27.690020797s) [upload=494645]: Next returned nil segment Dec 29 03:39:27.818949433 (564h10m27.690032469s) [upload=494645]: Waiting for error group managing segments... Dec 29 03:39:28.159200710 (564h10m28.030283755s) [noise=true , address=145.79.227.26:4653 , piece_id=WDQQCWBA4KEEBDUVDPGI6SKV2IIPZE57NYYB2DFH2BG4M3P3B2HQ, seg_pos={0 0}, upload=494645]: Done uploading piece. err: <nil> Dec 29 03:39:28.162010190 (564h10m28.033093235s) [noise=true , address=145.79.227.46:4601 , piece_id=XSW3FCQEJL3TE7TFP3LFVEGQA2XPW4TK5KZAJ2GDB2OHDHYTHHFQ, seg_pos={0 0}, upload=494645]: Done uploading piece. err: <nil> Dec 29 03:39:28.169628477 (564h10m28.040711512s) [noise=true , address=145.79.227.11:4730 , piece_id=VCXWUKAUMND4EZUJKLMBTI3I7GDH62JRILOFL4IVSRXLHQ2BCCIA, seg_pos={0 0}, upload=494645]: Done uploading piece. err: <nil> Dec 29 03:39:28.170299454 (564h10m28.0413825s) [noise=true , address=145.79.227.50:4374 , piece_id=7A5FFDBUYX4M63UBGLZAZJIRWZ3TOZBISSG35G7IXSALNY63YT2Q, seg_pos={0 0}, upload=494645]: Done uploading piece. err: <nil> Dec 29 03:39:28.170570491 (564h10m28.041653536s) [noise=true , address=145.79.227.2:4113 , piece_id=U2MVC6AWVKVR2P5LMDQBPN43SVO2BBSLCZJIXLK62VXBYMIJ4MOA, seg_pos={0 0}, upload=494645]: Done uploading piece. err: <nil> Dec 29 03:39:28.193625089 (564h10m28.064708134s) [noise=true , address=145.79.227.45:4230 , piece_id=DFMDFGRYGEJV5TXE6KFYNV6IU7PVULNOQCVKCQU7AJIHZJPI35MQ, seg_pos={0 0}, upload=494645]: Done uploading piece. err: <nil> Dec 29 03:39:28.200961199 (564h10m28.072044244s) [noise=true , address=145.79.227.47:4266 , piece_id=IMUISAR23BUUOJYS65UCZM5CUBN7YLA2K5BA44ZWCG46LEHBHHZA, seg_pos={0 0}, upload=494645]: Done uploading piece. err: <nil> Dec 29 03:39:28.206186509 (564h10m28.077269544s) [noise=true , address=145.79.227.19:4078 , piece_id=Z32HCP5MBSQRPO7YHLOTEARDTJDQTI4FS5P64SRDOI26U6YXFRTA, seg_pos={0 0}, upload=494645]: Done uploading piece. err: <nil> Dec 29 03:39:28.350158362 (564h10m28.221241397s) [noise=true , address=77.90.31.11:4554 , piece_id=DM32E2LNQNXEPSIRVFVW3RHNWXCH725MIWCR2VYDKE6JA2FDR6VA, seg_pos={0 0}, upload=494645]: Done uploading piece. err: <nil> Dec 29 03:39:28.509542051 (564h10m28.380625086s) [noise=true , address=45.91.116.11:4117 , piece_id=3YGZ3H4YC3QLM6NZPA3MZU7RB2VPNBY3IYVQEYGKT3HNGTPZFI6A, seg_pos={0 0}, upload=494645]: Done uploading piece. err: <nil> Dec 29 03:39:28.521022572 (564h10m28.392105617s) [noise=true , address=145.79.227.51:4411 , piece_id=LDPNKCXBIO5UBDEXUBAOOLY6342VBXTTNQB5P3HQ6JMI5HRQK7WQ, seg_pos={0 0}, upload=494645]: Done uploading piece. err: <nil> Dec 29 03:39:28.541207186 (564h10m28.412290231s) [noise=true , address=45.91.116.14:4191 , piece_id=3R6ASNVD23AVWVK4AHBMQJCH3TV2YSHM4SDZHK45AIO7ON2BM7BA, seg_pos={0 0}, upload=494645]: Done uploading piece. err: <nil> Dec 29 03:39:28.618079111 (564h10m28.489162146s) [noise=true , address=45.91.116.8:4045 , piece_id=I3ERFWTC5OWOKWGTU6ME2QJQFLROHHH5XUDTIGJGNP5YS5WMGQDA, seg_pos={0 0}, upload=494645]: Done uploading piece. err: <nil> Dec 29 03:39:28.618105992 (564h10m28.489189027s) [seg_pos={0 0}, upload=494645]: Segment reached optimal threshold of 13 pieces. Dec 29 03:39:28.618285093 (564h10m28.489368128s) [noise=true , address=45.91.116.9:4264 , piece_id=X35QO5ECUSQQ6J5E3XOZCGGSU3HNOIBGPTOOQO5QIH6MY2V54KRA, seg_pos={0 0}, upload=494645]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: write tcp 188.166.225.170:34026->45.91.116.9:4264: use of closed network connection; write tcp 188.166.225.170:34026->45.91.116.9:4264: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34026->45.91.116.9:4264: use of closed network connection Dec 29 03:39:28.618312606 (564h10m28.489395641s) [noise=true , address=45.91.116.9:4264 , piece_id=X35QO5ECUSQQ6J5E3XOZCGGSU3HNOIBGPTOOQO5QIH6MY2V54KRA, seg_pos={0 0}, upload=494645]: Long tail cancellation due to optimal threshold Dec 29 03:39:28.618418247 (564h10m28.489501292s) [noise=true , address=145.79.227.7:4295 , piece_id=7CZCFR2IZZ3FIXKNJFGH5XJTK7LOMR4XSXR3QRVFROFEOGYEMMOQ, seg_pos={0 0}, upload=494645]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT): protocol: expected piece hash; context canceled; EOF Dec 29 03:39:28.618423216 (564h10m28.489506251s) [noise=true , address=145.79.227.7:4295 , piece_id=7CZCFR2IZZ3FIXKNJFGH5XJTK7LOMR4XSXR3QRVFROFEOGYEMMOQ, seg_pos={0 0}, upload=494645]: Long tail cancellation due to optimal threshold Dec 29 03:39:28.619018039 (564h10m28.490101094s) [noise=true , address=77.90.31.10:4518 , piece_id=BE6Z5FDNUBQ4O2QFPQBRTYLNSENP3UVHHOVAPMC7WBEMKCCZ2QDQ, seg_pos={0 0}, upload=494645]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: expected piece hash; context canceled; EOF Dec 29 03:39:28.619037747 (564h10m28.490120782s) [noise=true , address=77.90.31.10:4518 , piece_id=BE6Z5FDNUBQ4O2QFPQBRTYLNSENP3UVHHOVAPMC7WBEMKCCZ2QDQ, seg_pos={0 0}, upload=494645]: Long tail cancellation due to optimal threshold Dec 29 03:39:28.619065129 (564h10m28.490148164s) [seg_pos={0 0}, upload=494645]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 03:39:28.619234201 (564h10m28.490317246s) [upload=494645]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 03:39:28.619241856 (564h10m28.490324891s) [upload=494645]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 03:39:28.904311268 (564h10m28.775394483s) [upload=494645]: Done uploading segments... Dec 29 03:39:28.904330285 (564h10m28.77541332s) [upload=494645]: Upload finished. err: <nil> Duration: 14.31517007s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....................................X...................X..] duration: 26.306375961s trace: https://jaeger.dev.storj.io/trace/2408171d336afafd FAIL (2025-12-29 03:40:07 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12Tz3HEHACmHU9Gp2W2vHVuF5PXNmdsa5N8nYhwt52HCTjggqRD, Piece ID: S3MEER2QCVCORYQNYNY2S22TC2H5HHKZW7VQJ54VL754UZTD42JQ) context deadline exceeded; piecestore: (Node ID: 1ve7KKYDbNo8Ma7wGVBH5x9z7zbPtPAPztmuQ9r6ijPpVgh5NX, Piece ID: S7T55BAWP53YJVHPW2GPLUM2VB3NBHZ2YD4UWBKKWCFHDAXAW3KQ) context deadline exceeded; piecestore: (Node ID: 1WVHPacX35JJ8PFzCs3UtGwsR4yVHXKmiqfGedK9KfXbHNJtmu, Piece ID: NYWKOTPLBZQIHGSQFMTT2QE6OQX7H5OKFVWJK3I65CWGGDSDDBMA) context deadline exceeded ======LOGS====== upload=494622]: Uploading piece... Dec 29 03:37:43.315384541 (564h8m43.186467596s) [noise=true , address=145.79.227.11:4762 , piece_id=2GQ5ASSBMUC52NCYRLBF3EOKYEA3G6DTIA6K56DE3OUQ4F5OLX2Q, seg_pos={0 0}, upload=494622]: Uploading piece... Dec 29 03:37:43.315899872 (564h8m43.186982907s) [noise=true , address=77.90.31.13:4649 , piece_id=S7T55BAWP53YJVHPW2GPLUM2VB3NBHZ2YD4UWBKKWCFHDAXAW3KQ, seg_pos={0 0}, upload=494622]: Uploading piece... Dec 29 03:37:43.316395165 (564h8m43.1874782s) [noise=true , address=77.90.31.18:4834 , piece_id=FQS6BFFJTORJPKJCINDZKXTY6GFRQTR3FZPWAEG7F2DNROUEPGJA, seg_pos={0 0}, upload=494622]: Uploading piece... Dec 29 03:37:43.316885961 (564h8m43.187969006s) [noise=true , address=193.149.164.143:4695 , piece_id=YF4EWRMS3GWEYPSA67GLG6EA5H7GZ55GTA3ISIEJ7RFEG2NDTLEQ, seg_pos={0 0}, upload=494622]: Uploading piece... Dec 29 03:37:43.317355795 (564h8m43.18843883s) [noise=true , address=145.79.227.7:4297 , piece_id=NNJ2KDDLNZHT4SGOUXBWBYD5OGW6MTGCAWZ63D5LH7BGM6KOJGDA, seg_pos={0 0}, upload=494622]: Uploading piece... Dec 29 03:37:43.317364631 (564h8m43.188447666s) [noise=true , address=145.79.227.33:4886 , piece_id=DB64UXSDC47SRTQX7PJGE66RCL7XFW3TGMCU3JOWVEJVGVHLYGWQ, seg_pos={0 0}, upload=494622]: Uploading piece... Dec 29 03:37:43.317689451 (564h8m43.188772486s) [noise=true , address=64.49.13.26:4950 , piece_id=HQ6BBIFQAD5RRN7FRWGDFHAVVMSDUWJUGTX7LBTA2TQJ3PNWKUHA, seg_pos={0 0}, upload=494622]: Uploading piece... Dec 29 03:37:43.317877448 (564h8m43.188960493s) [noise=true , address=45.91.116.10:4096 , piece_id=VIQK6EV67DONFZY77BOTWHY7POJEOE7FDHB6TKQMHVSHJJM35V7Q, seg_pos={0 0}, upload=494622]: Uploading piece... Dec 29 03:37:43.318315813 (564h8m43.189398858s) [noise=true , address=145.79.227.9:4388 , piece_id=DQU5TSAWTEEIBT272I76ZMGXD6FJT2QMXQISRDMWG5NOPORLBLLA, seg_pos={0 0}, upload=494622]: Uploading piece... Dec 29 03:37:43.318722317 (564h8m43.189805352s) [noise=true , address=77.90.31.12:4600 , piece_id=N4KXD5SYPYOKBXQRSL3UKNDDV7XB4KOKRYRYKGMMAHMAO6VVOEHQ, seg_pos={0 0}, upload=494622]: Uploading piece... Dec 29 03:37:43.319195248 (564h8m43.190278293s) [noise=true , address=45.91.116.8:4050 , piece_id=NYWKOTPLBZQIHGSQFMTT2QE6OQX7H5OKFVWJK3I65CWGGDSDDBMA, seg_pos={0 0}, upload=494622]: Uploading piece... Dec 29 03:37:43.319666174 (564h8m43.190749219s) [noise=true , address=77.90.31.21:4899 , piece_id=4MZKL4KVJF4DYGVPYGVOUHD6TYIDCRER33YPW6RG7KT6B6IXQEOA, seg_pos={0 0}, upload=494622]: Uploading piece... Dec 29 03:37:43.321557184 (564h8m43.192640229s) [noise=true , address=145.79.227.34:4909 , piece_id=UCGJPN357USFHIQ7II5GLRKNDE4H4HUVDQ7CB5R27C3VGRP2R3IA, seg_pos={0 0}, upload=494622]: Uploading piece... Dec 29 03:37:43.322231088 (564h8m43.193314133s) [noise=true , address=45.91.116.11:4139 , piece_id=S3MEER2QCVCORYQNYNY2S22TC2H5HHKZW7VQJ54VL754UZTD42JQ, seg_pos={0 0}, upload=494622]: Uploading piece... Dec 29 03:37:43.322851720 (564h8m43.193934755s) [noise=true , address=145.79.227.51:4444 , piece_id=EWEXXYYWF43J7FG6NPSJ2PICHTRPY533L4DQQX2CSW5IMJ6CB2PQ, seg_pos={0 0}, upload=494622]: Uploading piece... Dec 29 03:37:43.323435972 (564h8m43.194519007s) [noise=true , address=145.79.227.40:4068 , piece_id=4QOJWRVEE3ERM2AHP764XSTE7ZYHUUJ2ENBVXRS5UDIGKZOK7YKA, seg_pos={0 0}, upload=494622]: Uploading piece... Dec 29 03:37:43.323901118 (564h8m43.194984153s) [noise=true , address=145.79.227.30:4790 , piece_id=2TSIHAVWOZPOVKSTG2ZXBBEFEMMMRLMOWHM47TDANARFPDJM3NPQ, seg_pos={0 0}, upload=494622]: Uploading piece... Dec 29 03:37:44.620573279 (564h8m44.491656314s) [upload=494622]: Next returned nil segment Dec 29 03:37:44.620593617 (564h8m44.491676652s) [upload=494622]: Waiting for error group managing segments... Dec 29 03:37:45.178264102 (564h8m45.049347137s) [noise=true , address=145.79.227.11:4762 , piece_id=2GQ5ASSBMUC52NCYRLBF3EOKYEA3G6DTIA6K56DE3OUQ4F5OLX2Q, seg_pos={0 0}, upload=494622]: Done uploading piece. err: <nil> Dec 29 03:37:45.190189951 (564h8m45.061272986s) [noise=true , address=145.79.227.51:4444 , piece_id=EWEXXYYWF43J7FG6NPSJ2PICHTRPY533L4DQQX2CSW5IMJ6CB2PQ, seg_pos={0 0}, upload=494622]: Done uploading piece. err: <nil> Dec 29 03:37:45.199163818 (564h8m45.070246853s) [noise=true , address=145.79.227.7:4297 , piece_id=NNJ2KDDLNZHT4SGOUXBWBYD5OGW6MTGCAWZ63D5LH7BGM6KOJGDA, seg_pos={0 0}, upload=494622]: Done uploading piece. err: <nil> Dec 29 03:37:45.448727893 (564h8m45.319810928s) [noise=true , address=45.91.116.11:4139 , piece_id=S3MEER2QCVCORYQNYNY2S22TC2H5HHKZW7VQJ54VL754UZTD42JQ, seg_pos={0 0}, upload=494622]: Done uploading piece. err: <nil> Dec 29 03:37:45.459119140 (564h8m45.330202175s) [noise=true , address=77.90.31.13:4649 , piece_id=S7T55BAWP53YJVHPW2GPLUM2VB3NBHZ2YD4UWBKKWCFHDAXAW3KQ, seg_pos={0 0}, upload=494622]: Done uploading piece. err: <nil> Dec 29 03:37:45.471609714 (564h8m45.342692759s) [noise=true , address=45.91.116.8:4050 , piece_id=NYWKOTPLBZQIHGSQFMTT2QE6OQX7H5OKFVWJK3I65CWGGDSDDBMA, seg_pos={0 0}, upload=494622]: Done uploading piece. err: <nil> Dec 29 03:37:45.489098314 (564h8m45.36018136s) [noise=true , address=77.90.31.21:4899 , piece_id=4MZKL4KVJF4DYGVPYGVOUHD6TYIDCRER33YPW6RG7KT6B6IXQEOA, seg_pos={0 0}, upload=494622]: Done uploading piece. err: <nil> Dec 29 03:37:45.489726060 (564h8m45.360809105s) [noise=true , address=77.90.31.11:4574 , piece_id=3TP2ME5NTXVK45B34W5U4DESLFJLOJEPRQOJLEEPN2PIMYS2IDWQ, seg_pos={0 0}, upload=494622]: Done uploading piece. err: <nil> Dec 29 03:37:45.779669865 (564h8m45.65075291s) [noise=true , address=145.79.227.9:4388 , piece_id=DQU5TSAWTEEIBT272I76ZMGXD6FJT2QMXQISRDMWG5NOPORLBLLA, seg_pos={0 0}, upload=494622]: Done uploading piece. err: <nil> Dec 29 03:37:45.817564973 (564h8m45.688648008s) [noise=true , address=77.90.31.18:4834 , piece_id=FQS6BFFJTORJPKJCINDZKXTY6GFRQTR3FZPWAEG7F2DNROUEPGJA, seg_pos={0 0}, upload=494622]: Done uploading piece. err: <nil> Dec 29 03:37:46.001090802 (564h8m45.872173857s) [noise=true , address=193.149.164.143:4695 , piece_id=YF4EWRMS3GWEYPSA67GLG6EA5H7GZ55GTA3ISIEJ7RFEG2NDTLEQ, seg_pos={0 0}, upload=494622]: Done uploading piece. err: <nil> Dec 29 03:37:46.129753764 (564h8m46.000836819s) [noise=true , address=145.79.227.33:4886 , piece_id=DB64UXSDC47SRTQX7PJGE66RCL7XFW3TGMCU3JOWVEJVGVHLYGWQ, seg_pos={0 0}, upload=494622]: Done uploading piece. err: <nil> Dec 29 03:37:46.163280064 (564h8m46.034363109s) [noise=true , address=145.79.227.34:4909 , piece_id=UCGJPN357USFHIQ7II5GLRKNDE4H4HUVDQ7CB5R27C3VGRP2R3IA, seg_pos={0 0}, upload=494622]: Done uploading piece. err: <nil> Dec 29 03:37:46.167742170 (564h8m46.038825205s) [noise=true , address=145.79.227.30:4790 , piece_id=2TSIHAVWOZPOVKSTG2ZXBBEFEMMMRLMOWHM47TDANARFPDJM3NPQ, seg_pos={0 0}, upload=494622]: Done uploading piece. err: <nil> Dec 29 03:37:46.167764403 (564h8m46.038847448s) [seg_pos={0 0}, upload=494622]: Segment reached optimal threshold of 14 pieces. Dec 29 03:37:46.167996534 (564h8m46.039079569s) [noise=true , address=45.91.116.10:4096 , piece_id=VIQK6EV67DONFZY77BOTWHY7POJEOE7FDHB6TKQMHVSHJJM35V7Q, seg_pos={0 0}, upload=494622]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12MZX1GwZGrEi2q44dPiYrwMoum5A2caZtXTS3qiLNcoyHhon7p): protocol: expected piece hash; context canceled; EOF Dec 29 03:37:46.168005812 (564h8m46.039088857s) [noise=true , address=45.91.116.10:4096 , piece_id=VIQK6EV67DONFZY77BOTWHY7POJEOE7FDHB6TKQMHVSHJJM35V7Q, seg_pos={0 0}, upload=494622]: Long tail cancellation due to optimal threshold Dec 29 03:37:46.168120160 (564h8m46.039203195s) [noise=true , address=64.49.13.26:4950 , piece_id=HQ6BBIFQAD5RRN7FRWGDFHAVVMSDUWJUGTX7LBTA2TQJ3PNWKUHA, seg_pos={0 0}, upload=494622]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14Tv25fLSqTnZQiDzrYqKCSbdt6ep3upgqD2Se8QSRuQFynkSN): protocol: expected piece hash; context canceled; EOF Dec 29 03:37:46.168136090 (564h8m46.039219125s) [noise=true , address=64.49.13.26:4950 , piece_id=HQ6BBIFQAD5RRN7FRWGDFHAVVMSDUWJUGTX7LBTA2TQJ3PNWKUHA, seg_pos={0 0}, upload=494622]: Long tail cancellation due to optimal threshold Dec 29 03:37:46.168274995 (564h8m46.03935804s) [noise=true , address=77.90.31.12:4600 , piece_id=N4KXD5SYPYOKBXQRSL3UKNDDV7XB4KOKRYRYKGMMAHMAO6VVOEHQ, seg_pos={0 0}, upload=494622]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PKT2EcU6jAe6oUUdiTcwxGQsFa36uEmown77SemqQMM5Xw5nc): protocol: expected piece hash; context canceled; EOF Dec 29 03:37:46.168283491 (564h8m46.039366536s) [noise=true , address=77.90.31.12:4600 , piece_id=N4KXD5SYPYOKBXQRSL3UKNDDV7XB4KOKRYRYKGMMAHMAO6VVOEHQ, seg_pos={0 0}, upload=494622]: Long tail cancellation due to optimal threshold Dec 29 03:37:46.168394032 (564h8m46.039477067s) [noise=true , address=145.79.227.40:4068 , piece_id=4QOJWRVEE3ERM2AHP764XSTE7ZYHUUJ2ENBVXRS5UDIGKZOK7YKA, seg_pos={0 0}, upload=494622]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Lh9dYpHCxbs7gpZ4Xt7os5Gm5Nco6jrNW1V1FHddVUEMkFfPY): protocol: expected piece hash; context canceled; EOF Dec 29 03:37:46.168403810 (564h8m46.039486845s) [noise=true , address=145.79.227.40:4068 , piece_id=4QOJWRVEE3ERM2AHP764XSTE7ZYHUUJ2ENBVXRS5UDIGKZOK7YKA, seg_pos={0 0}, upload=494622]: Long tail cancellation due to optimal threshold Dec 29 03:37:46.168432725 (564h8m46.03951577s) [seg_pos={0 0}, upload=494622]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 03:37:46.168576960 (564h8m46.039659995s) [upload=494622]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 03:37:46.168584033 (564h8m46.039667068s) [upload=494622]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 03:37:46.466683658 (564h8m46.337766723s) [upload=494622]: Done uploading segments... Dec 29 03:37:46.466712052 (564h8m46.337795087s) [upload=494622]: Upload finished. err: <nil> Duration: 26.306405087s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 5 out of 60 is failed (threshold=6) [.................................XX................X......XX] duration: 5.001694934s trace: https://jaeger.dev.storj.io/trace/6de9f17093e778ca FAIL (2025-12-29 03:40:04 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001732826s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 29 out of 60 is failed (threshold=3) [X..XX.....XX.X.....X.X..XXXX..XXX.XX...X.X.....XXX...XXXXXXX] duration: 30.003037734s trace: https://jaeger.dev.storj.io/trace/5d868317ce787ff0 FAIL (2025-12-29 03:39:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, Piece ID: 3R6ASNVD23AVWVK4AHBMQJCH3TV2YSHM4SDZHK45AIO7ON2BM7BA) context deadline exceeded; piecestore: (Node ID: 1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, Piece ID: DM32E2LNQNXEPSIRVFVW3RHNWXCH725MIWCR2VYDKE6JA2FDR6VA) context deadline exceeded; piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: I3ERFWTC5OWOKWGTU6ME2QJQFLROHHH5XUDTIGJGNP5YS5WMGQDA) context deadline exceeded ======LOGS====== 29 03:39:25.931217811 (564h10m25.802300846s) [upload=494645]: Starting upload Dec 29 03:39:25.931273377 (564h10m25.802356412s) [upload=494645]: Done starting upload Dec 29 03:39:25.931469391 (564h10m25.802552426s) [upload=494645]: Uploading segments... Dec 29 03:39:25.931711250 (564h10m25.802794296s) [upload=494645]: Got next segment. Inline: false Dec 29 03:39:25.931762458 (564h10m25.802845503s) [upload=494645]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 03:39:25.931769441 (564h10m25.802852897s) [upload=494645]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 03:39:26.174866223 (564h10m26.045949258s) [seg_pos={0 0}, upload=494645]: Begin upload segment... Dec 29 03:39:26.174911159 (564h10m26.045994204s) [seg_pos={0 0}, upload=494645]: Done begin upload segment. Dec 29 03:39:26.174922570 (564h10m26.046005615s) [noise=true , address=145.79.227.51:4411 , piece_id=LDPNKCXBIO5UBDEXUBAOOLY6342VBXTTNQB5P3HQ6JMI5HRQK7WQ, seg_pos={0 0}, upload=494645]: Uploading piece... Dec 29 03:39:26.174937790 (564h10m26.046020835s) [noise=true , address=145.79.227.26:4653 , piece_id=WDQQCWBA4KEEBDUVDPGI6SKV2IIPZE57NYYB2DFH2BG4M3P3B2HQ, seg_pos={0 0}, upload=494645]: Uploading piece... Dec 29 03:39:26.175624657 (564h10m26.046707692s) [noise=true , address=45.91.116.9:4264 , piece_id=X35QO5ECUSQQ6J5E3XOZCGGSU3HNOIBGPTOOQO5QIH6MY2V54KRA, seg_pos={0 0}, upload=494645]: Uploading piece... Dec 29 03:39:26.175941531 (564h10m26.047024566s) [noise=true , address=77.90.31.11:4554 , piece_id=DM32E2LNQNXEPSIRVFVW3RHNWXCH725MIWCR2VYDKE6JA2FDR6VA, seg_pos={0 0}, upload=494645]: Uploading piece... Dec 29 03:39:26.176222386 (564h10m26.047305421s) [noise=true , address=145.79.227.11:4730 , piece_id=VCXWUKAUMND4EZUJKLMBTI3I7GDH62JRILOFL4IVSRXLHQ2BCCIA, seg_pos={0 0}, upload=494645]: Uploading piece... Dec 29 03:39:26.176509432 (564h10m26.047592477s) [noise=true , address=145.79.227.7:4295 , piece_id=7CZCFR2IZZ3FIXKNJFGH5XJTK7LOMR4XSXR3QRVFROFEOGYEMMOQ, seg_pos={0 0}, upload=494645]: Uploading piece... Dec 29 03:39:26.176782362 (564h10m26.047865397s) [noise=true , address=145.79.227.19:4078 , piece_id=Z32HCP5MBSQRPO7YHLOTEARDTJDQTI4FS5P64SRDOI26U6YXFRTA, seg_pos={0 0}, upload=494645]: Uploading piece... Dec 29 03:39:26.177049660 (564h10m26.048132705s) [noise=true , address=145.79.227.47:4266 , piece_id=IMUISAR23BUUOJYS65UCZM5CUBN7YLA2K5BA44ZWCG46LEHBHHZA, seg_pos={0 0}, upload=494645]: Uploading piece... Dec 29 03:39:26.177387153 (564h10m26.048470188s) [noise=true , address=145.79.227.2:4113 , piece_id=U2MVC6AWVKVR2P5LMDQBPN43SVO2BBSLCZJIXLK62VXBYMIJ4MOA, seg_pos={0 0}, upload=494645]: Uploading piece... Dec 29 03:39:26.177834585 (564h10m26.04891763s) [noise=true , address=45.91.116.8:4045 , piece_id=I3ERFWTC5OWOKWGTU6ME2QJQFLROHHH5XUDTIGJGNP5YS5WMGQDA, seg_pos={0 0}, upload=494645]: Uploading piece... Dec 29 03:39:26.178904823 (564h10m26.049987859s) [noise=true , address=45.91.116.11:4117 , piece_id=3YGZ3H4YC3QLM6NZPA3MZU7RB2VPNBY3IYVQEYGKT3HNGTPZFI6A, seg_pos={0 0}, upload=494645]: Uploading piece... Dec 29 03:39:26.179245491 (564h10m26.050328526s) [noise=true , address=145.79.227.45:4230 , piece_id=DFMDFGRYGEJV5TXE6KFYNV6IU7PVULNOQCVKCQU7AJIHZJPI35MQ, seg_pos={0 0}, upload=494645]: Uploading piece... Dec 29 03:39:26.180029956 (564h10m26.051112991s) [noise=true , address=145.79.227.50:4374 , piece_id=7A5FFDBUYX4M63UBGLZAZJIRWZ3TOZBISSG35G7IXSALNY63YT2Q, seg_pos={0 0}, upload=494645]: Uploading piece... Dec 29 03:39:26.180527964 (564h10m26.051610999s) [noise=true , address=45.91.116.14:4191 , piece_id=3R6ASNVD23AVWVK4AHBMQJCH3TV2YSHM4SDZHK45AIO7ON2BM7BA, seg_pos={0 0}, upload=494645]: Uploading piece... Dec 29 03:39:26.181227546 (564h10m26.052310591s) [noise=true , address=145.79.227.46:4601 , piece_id=XSW3FCQEJL3TE7TFP3LFVEGQA2XPW4TK5KZAJ2GDB2OHDHYTHHFQ, seg_pos={0 0}, upload=494645]: Uploading piece... Dec 29 03:39:26.182419296 (564h10m26.053502341s) [noise=true , address=77.90.31.10:4518 , piece_id=BE6Z5FDNUBQ4O2QFPQBRTYLNSENP3UVHHOVAPMC7WBEMKCCZ2QDQ, seg_pos={0 0}, upload=494645]: Uploading piece... Dec 29 03:39:27.818937762 (564h10m27.690020797s) [upload=494645]: Next returned nil segment Dec 29 03:39:27.818949433 (564h10m27.690032469s) [upload=494645]: Waiting for error group managing segments... Dec 29 03:39:28.159200710 (564h10m28.030283755s) [noise=true , address=145.79.227.26:4653 , piece_id=WDQQCWBA4KEEBDUVDPGI6SKV2IIPZE57NYYB2DFH2BG4M3P3B2HQ, seg_pos={0 0}, upload=494645]: Done uploading piece. err: <nil> Dec 29 03:39:28.162010190 (564h10m28.033093235s) [noise=true , address=145.79.227.46:4601 , piece_id=XSW3FCQEJL3TE7TFP3LFVEGQA2XPW4TK5KZAJ2GDB2OHDHYTHHFQ, seg_pos={0 0}, upload=494645]: Done uploading piece. err: <nil> Dec 29 03:39:28.169628477 (564h10m28.040711512s) [noise=true , address=145.79.227.11:4730 , piece_id=VCXWUKAUMND4EZUJKLMBTI3I7GDH62JRILOFL4IVSRXLHQ2BCCIA, seg_pos={0 0}, upload=494645]: Done uploading piece. err: <nil> Dec 29 03:39:28.170299454 (564h10m28.0413825s) [noise=true , address=145.79.227.50:4374 , piece_id=7A5FFDBUYX4M63UBGLZAZJIRWZ3TOZBISSG35G7IXSALNY63YT2Q, seg_pos={0 0}, upload=494645]: Done uploading piece. err: <nil> Dec 29 03:39:28.170570491 (564h10m28.041653536s) [noise=true , address=145.79.227.2:4113 , piece_id=U2MVC6AWVKVR2P5LMDQBPN43SVO2BBSLCZJIXLK62VXBYMIJ4MOA, seg_pos={0 0}, upload=494645]: Done uploading piece. err: <nil> Dec 29 03:39:28.193625089 (564h10m28.064708134s) [noise=true , address=145.79.227.45:4230 , piece_id=DFMDFGRYGEJV5TXE6KFYNV6IU7PVULNOQCVKCQU7AJIHZJPI35MQ, seg_pos={0 0}, upload=494645]: Done uploading piece. err: <nil> Dec 29 03:39:28.200961199 (564h10m28.072044244s) [noise=true , address=145.79.227.47:4266 , piece_id=IMUISAR23BUUOJYS65UCZM5CUBN7YLA2K5BA44ZWCG46LEHBHHZA, seg_pos={0 0}, upload=494645]: Done uploading piece. err: <nil> Dec 29 03:39:28.206186509 (564h10m28.077269544s) [noise=true , address=145.79.227.19:4078 , piece_id=Z32HCP5MBSQRPO7YHLOTEARDTJDQTI4FS5P64SRDOI26U6YXFRTA, seg_pos={0 0}, upload=494645]: Done uploading piece. err: <nil> Dec 29 03:39:28.350158362 (564h10m28.221241397s) [noise=true , address=77.90.31.11:4554 , piece_id=DM32E2LNQNXEPSIRVFVW3RHNWXCH725MIWCR2VYDKE6JA2FDR6VA, seg_pos={0 0}, upload=494645]: Done uploading piece. err: <nil> Dec 29 03:39:28.509542051 (564h10m28.380625086s) [noise=true , address=45.91.116.11:4117 , piece_id=3YGZ3H4YC3QLM6NZPA3MZU7RB2VPNBY3IYVQEYGKT3HNGTPZFI6A, seg_pos={0 0}, upload=494645]: Done uploading piece. err: <nil> Dec 29 03:39:28.521022572 (564h10m28.392105617s) [noise=true , address=145.79.227.51:4411 , piece_id=LDPNKCXBIO5UBDEXUBAOOLY6342VBXTTNQB5P3HQ6JMI5HRQK7WQ, seg_pos={0 0}, upload=494645]: Done uploading piece. err: <nil> Dec 29 03:39:28.541207186 (564h10m28.412290231s) [noise=true , address=45.91.116.14:4191 , piece_id=3R6ASNVD23AVWVK4AHBMQJCH3TV2YSHM4SDZHK45AIO7ON2BM7BA, seg_pos={0 0}, upload=494645]: Done uploading piece. err: <nil> Dec 29 03:39:28.618079111 (564h10m28.489162146s) [noise=true , address=45.91.116.8:4045 , piece_id=I3ERFWTC5OWOKWGTU6ME2QJQFLROHHH5XUDTIGJGNP5YS5WMGQDA, seg_pos={0 0}, upload=494645]: Done uploading piece. err: <nil> Dec 29 03:39:28.618105992 (564h10m28.489189027s) [seg_pos={0 0}, upload=494645]: Segment reached optimal threshold of 13 pieces. Dec 29 03:39:28.618285093 (564h10m28.489368128s) [noise=true , address=45.91.116.9:4264 , piece_id=X35QO5ECUSQQ6J5E3XOZCGGSU3HNOIBGPTOOQO5QIH6MY2V54KRA, seg_pos={0 0}, upload=494645]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: write tcp 188.166.225.170:34026->45.91.116.9:4264: use of closed network connection; write tcp 188.166.225.170:34026->45.91.116.9:4264: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34026->45.91.116.9:4264: use of closed network connection Dec 29 03:39:28.618312606 (564h10m28.489395641s) [noise=true , address=45.91.116.9:4264 , piece_id=X35QO5ECUSQQ6J5E3XOZCGGSU3HNOIBGPTOOQO5QIH6MY2V54KRA, seg_pos={0 0}, upload=494645]: Long tail cancellation due to optimal threshold Dec 29 03:39:28.618418247 (564h10m28.489501292s) [noise=true , address=145.79.227.7:4295 , piece_id=7CZCFR2IZZ3FIXKNJFGH5XJTK7LOMR4XSXR3QRVFROFEOGYEMMOQ, seg_pos={0 0}, upload=494645]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT): protocol: expected piece hash; context canceled; EOF Dec 29 03:39:28.618423216 (564h10m28.489506251s) [noise=true , address=145.79.227.7:4295 , piece_id=7CZCFR2IZZ3FIXKNJFGH5XJTK7LOMR4XSXR3QRVFROFEOGYEMMOQ, seg_pos={0 0}, upload=494645]: Long tail cancellation due to optimal threshold Dec 29 03:39:28.619018039 (564h10m28.490101094s) [noise=true , address=77.90.31.10:4518 , piece_id=BE6Z5FDNUBQ4O2QFPQBRTYLNSENP3UVHHOVAPMC7WBEMKCCZ2QDQ, seg_pos={0 0}, upload=494645]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: expected piece hash; context canceled; EOF Dec 29 03:39:28.619037747 (564h10m28.490120782s) [noise=true , address=77.90.31.10:4518 , piece_id=BE6Z5FDNUBQ4O2QFPQBRTYLNSENP3UVHHOVAPMC7WBEMKCCZ2QDQ, seg_pos={0 0}, upload=494645]: Long tail cancellation due to optimal threshold Dec 29 03:39:28.619065129 (564h10m28.490148164s) [seg_pos={0 0}, upload=494645]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 03:39:28.619234201 (564h10m28.490317246s) [upload=494645]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 03:39:28.619241856 (564h10m28.490324891s) [upload=494645]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 03:39:28.904311268 (564h10m28.775394483s) [upload=494645]: Done uploading segments... Dec 29 03:39:28.904330285 (564h10m28.77541332s) [upload=494645]: Upload finished. err: <nil> Duration: 30.00306125s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [......................................X...................X.] duration: 33.217277208s trace: https://jaeger.dev.storj.io/trace/2408171d336afafd FAIL (2025-12-29 03:39:15 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12Tz3HEHACmHU9Gp2W2vHVuF5PXNmdsa5N8nYhwt52HCTjggqRD, Piece ID: S3MEER2QCVCORYQNYNY2S22TC2H5HHKZW7VQJ54VL754UZTD42JQ) context deadline exceeded; piecestore: (Node ID: 1ve7KKYDbNo8Ma7wGVBH5x9z7zbPtPAPztmuQ9r6ijPpVgh5NX, Piece ID: S7T55BAWP53YJVHPW2GPLUM2VB3NBHZ2YD4UWBKKWCFHDAXAW3KQ) context deadline exceeded; piecestore: (Node ID: 1WVHPacX35JJ8PFzCs3UtGwsR4yVHXKmiqfGedK9KfXbHNJtmu, Piece ID: NYWKOTPLBZQIHGSQFMTT2QE6OQX7H5OKFVWJK3I65CWGGDSDDBMA) context deadline exceeded ======LOGS====== upload=494622]: Uploading piece... Dec 29 03:37:43.315384541 (564h8m43.186467596s) [noise=true , address=145.79.227.11:4762 , piece_id=2GQ5ASSBMUC52NCYRLBF3EOKYEA3G6DTIA6K56DE3OUQ4F5OLX2Q, seg_pos={0 0}, upload=494622]: Uploading piece... Dec 29 03:37:43.315899872 (564h8m43.186982907s) [noise=true , address=77.90.31.13:4649 , piece_id=S7T55BAWP53YJVHPW2GPLUM2VB3NBHZ2YD4UWBKKWCFHDAXAW3KQ, seg_pos={0 0}, upload=494622]: Uploading piece... Dec 29 03:37:43.316395165 (564h8m43.1874782s) [noise=true , address=77.90.31.18:4834 , piece_id=FQS6BFFJTORJPKJCINDZKXTY6GFRQTR3FZPWAEG7F2DNROUEPGJA, seg_pos={0 0}, upload=494622]: Uploading piece... Dec 29 03:37:43.316885961 (564h8m43.187969006s) [noise=true , address=193.149.164.143:4695 , piece_id=YF4EWRMS3GWEYPSA67GLG6EA5H7GZ55GTA3ISIEJ7RFEG2NDTLEQ, seg_pos={0 0}, upload=494622]: Uploading piece... Dec 29 03:37:43.317355795 (564h8m43.18843883s) [noise=true , address=145.79.227.7:4297 , piece_id=NNJ2KDDLNZHT4SGOUXBWBYD5OGW6MTGCAWZ63D5LH7BGM6KOJGDA, seg_pos={0 0}, upload=494622]: Uploading piece... Dec 29 03:37:43.317364631 (564h8m43.188447666s) [noise=true , address=145.79.227.33:4886 , piece_id=DB64UXSDC47SRTQX7PJGE66RCL7XFW3TGMCU3JOWVEJVGVHLYGWQ, seg_pos={0 0}, upload=494622]: Uploading piece... Dec 29 03:37:43.317689451 (564h8m43.188772486s) [noise=true , address=64.49.13.26:4950 , piece_id=HQ6BBIFQAD5RRN7FRWGDFHAVVMSDUWJUGTX7LBTA2TQJ3PNWKUHA, seg_pos={0 0}, upload=494622]: Uploading piece... Dec 29 03:37:43.317877448 (564h8m43.188960493s) [noise=true , address=45.91.116.10:4096 , piece_id=VIQK6EV67DONFZY77BOTWHY7POJEOE7FDHB6TKQMHVSHJJM35V7Q, seg_pos={0 0}, upload=494622]: Uploading piece... Dec 29 03:37:43.318315813 (564h8m43.189398858s) [noise=true , address=145.79.227.9:4388 , piece_id=DQU5TSAWTEEIBT272I76ZMGXD6FJT2QMXQISRDMWG5NOPORLBLLA, seg_pos={0 0}, upload=494622]: Uploading piece... Dec 29 03:37:43.318722317 (564h8m43.189805352s) [noise=true , address=77.90.31.12:4600 , piece_id=N4KXD5SYPYOKBXQRSL3UKNDDV7XB4KOKRYRYKGMMAHMAO6VVOEHQ, seg_pos={0 0}, upload=494622]: Uploading piece... Dec 29 03:37:43.319195248 (564h8m43.190278293s) [noise=true , address=45.91.116.8:4050 , piece_id=NYWKOTPLBZQIHGSQFMTT2QE6OQX7H5OKFVWJK3I65CWGGDSDDBMA, seg_pos={0 0}, upload=494622]: Uploading piece... Dec 29 03:37:43.319666174 (564h8m43.190749219s) [noise=true , address=77.90.31.21:4899 , piece_id=4MZKL4KVJF4DYGVPYGVOUHD6TYIDCRER33YPW6RG7KT6B6IXQEOA, seg_pos={0 0}, upload=494622]: Uploading piece... Dec 29 03:37:43.321557184 (564h8m43.192640229s) [noise=true , address=145.79.227.34:4909 , piece_id=UCGJPN357USFHIQ7II5GLRKNDE4H4HUVDQ7CB5R27C3VGRP2R3IA, seg_pos={0 0}, upload=494622]: Uploading piece... Dec 29 03:37:43.322231088 (564h8m43.193314133s) [noise=true , address=45.91.116.11:4139 , piece_id=S3MEER2QCVCORYQNYNY2S22TC2H5HHKZW7VQJ54VL754UZTD42JQ, seg_pos={0 0}, upload=494622]: Uploading piece... Dec 29 03:37:43.322851720 (564h8m43.193934755s) [noise=true , address=145.79.227.51:4444 , piece_id=EWEXXYYWF43J7FG6NPSJ2PICHTRPY533L4DQQX2CSW5IMJ6CB2PQ, seg_pos={0 0}, upload=494622]: Uploading piece... Dec 29 03:37:43.323435972 (564h8m43.194519007s) [noise=true , address=145.79.227.40:4068 , piece_id=4QOJWRVEE3ERM2AHP764XSTE7ZYHUUJ2ENBVXRS5UDIGKZOK7YKA, seg_pos={0 0}, upload=494622]: Uploading piece... Dec 29 03:37:43.323901118 (564h8m43.194984153s) [noise=true , address=145.79.227.30:4790 , piece_id=2TSIHAVWOZPOVKSTG2ZXBBEFEMMMRLMOWHM47TDANARFPDJM3NPQ, seg_pos={0 0}, upload=494622]: Uploading piece... Dec 29 03:37:44.620573279 (564h8m44.491656314s) [upload=494622]: Next returned nil segment Dec 29 03:37:44.620593617 (564h8m44.491676652s) [upload=494622]: Waiting for error group managing segments... Dec 29 03:37:45.178264102 (564h8m45.049347137s) [noise=true , address=145.79.227.11:4762 , piece_id=2GQ5ASSBMUC52NCYRLBF3EOKYEA3G6DTIA6K56DE3OUQ4F5OLX2Q, seg_pos={0 0}, upload=494622]: Done uploading piece. err: <nil> Dec 29 03:37:45.190189951 (564h8m45.061272986s) [noise=true , address=145.79.227.51:4444 , piece_id=EWEXXYYWF43J7FG6NPSJ2PICHTRPY533L4DQQX2CSW5IMJ6CB2PQ, seg_pos={0 0}, upload=494622]: Done uploading piece. err: <nil> Dec 29 03:37:45.199163818 (564h8m45.070246853s) [noise=true , address=145.79.227.7:4297 , piece_id=NNJ2KDDLNZHT4SGOUXBWBYD5OGW6MTGCAWZ63D5LH7BGM6KOJGDA, seg_pos={0 0}, upload=494622]: Done uploading piece. err: <nil> Dec 29 03:37:45.448727893 (564h8m45.319810928s) [noise=true , address=45.91.116.11:4139 , piece_id=S3MEER2QCVCORYQNYNY2S22TC2H5HHKZW7VQJ54VL754UZTD42JQ, seg_pos={0 0}, upload=494622]: Done uploading piece. err: <nil> Dec 29 03:37:45.459119140 (564h8m45.330202175s) [noise=true , address=77.90.31.13:4649 , piece_id=S7T55BAWP53YJVHPW2GPLUM2VB3NBHZ2YD4UWBKKWCFHDAXAW3KQ, seg_pos={0 0}, upload=494622]: Done uploading piece. err: <nil> Dec 29 03:37:45.471609714 (564h8m45.342692759s) [noise=true , address=45.91.116.8:4050 , piece_id=NYWKOTPLBZQIHGSQFMTT2QE6OQX7H5OKFVWJK3I65CWGGDSDDBMA, seg_pos={0 0}, upload=494622]: Done uploading piece. err: <nil> Dec 29 03:37:45.489098314 (564h8m45.36018136s) [noise=true , address=77.90.31.21:4899 , piece_id=4MZKL4KVJF4DYGVPYGVOUHD6TYIDCRER33YPW6RG7KT6B6IXQEOA, seg_pos={0 0}, upload=494622]: Done uploading piece. err: <nil> Dec 29 03:37:45.489726060 (564h8m45.360809105s) [noise=true , address=77.90.31.11:4574 , piece_id=3TP2ME5NTXVK45B34W5U4DESLFJLOJEPRQOJLEEPN2PIMYS2IDWQ, seg_pos={0 0}, upload=494622]: Done uploading piece. err: <nil> Dec 29 03:37:45.779669865 (564h8m45.65075291s) [noise=true , address=145.79.227.9:4388 , piece_id=DQU5TSAWTEEIBT272I76ZMGXD6FJT2QMXQISRDMWG5NOPORLBLLA, seg_pos={0 0}, upload=494622]: Done uploading piece. err: <nil> Dec 29 03:37:45.817564973 (564h8m45.688648008s) [noise=true , address=77.90.31.18:4834 , piece_id=FQS6BFFJTORJPKJCINDZKXTY6GFRQTR3FZPWAEG7F2DNROUEPGJA, seg_pos={0 0}, upload=494622]: Done uploading piece. err: <nil> Dec 29 03:37:46.001090802 (564h8m45.872173857s) [noise=true , address=193.149.164.143:4695 , piece_id=YF4EWRMS3GWEYPSA67GLG6EA5H7GZ55GTA3ISIEJ7RFEG2NDTLEQ, seg_pos={0 0}, upload=494622]: Done uploading piece. err: <nil> Dec 29 03:37:46.129753764 (564h8m46.000836819s) [noise=true , address=145.79.227.33:4886 , piece_id=DB64UXSDC47SRTQX7PJGE66RCL7XFW3TGMCU3JOWVEJVGVHLYGWQ, seg_pos={0 0}, upload=494622]: Done uploading piece. err: <nil> Dec 29 03:37:46.163280064 (564h8m46.034363109s) [noise=true , address=145.79.227.34:4909 , piece_id=UCGJPN357USFHIQ7II5GLRKNDE4H4HUVDQ7CB5R27C3VGRP2R3IA, seg_pos={0 0}, upload=494622]: Done uploading piece. err: <nil> Dec 29 03:37:46.167742170 (564h8m46.038825205s) [noise=true , address=145.79.227.30:4790 , piece_id=2TSIHAVWOZPOVKSTG2ZXBBEFEMMMRLMOWHM47TDANARFPDJM3NPQ, seg_pos={0 0}, upload=494622]: Done uploading piece. err: <nil> Dec 29 03:37:46.167764403 (564h8m46.038847448s) [seg_pos={0 0}, upload=494622]: Segment reached optimal threshold of 14 pieces. Dec 29 03:37:46.167996534 (564h8m46.039079569s) [noise=true , address=45.91.116.10:4096 , piece_id=VIQK6EV67DONFZY77BOTWHY7POJEOE7FDHB6TKQMHVSHJJM35V7Q, seg_pos={0 0}, upload=494622]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12MZX1GwZGrEi2q44dPiYrwMoum5A2caZtXTS3qiLNcoyHhon7p): protocol: expected piece hash; context canceled; EOF Dec 29 03:37:46.168005812 (564h8m46.039088857s) [noise=true , address=45.91.116.10:4096 , piece_id=VIQK6EV67DONFZY77BOTWHY7POJEOE7FDHB6TKQMHVSHJJM35V7Q, seg_pos={0 0}, upload=494622]: Long tail cancellation due to optimal threshold Dec 29 03:37:46.168120160 (564h8m46.039203195s) [noise=true , address=64.49.13.26:4950 , piece_id=HQ6BBIFQAD5RRN7FRWGDFHAVVMSDUWJUGTX7LBTA2TQJ3PNWKUHA, seg_pos={0 0}, upload=494622]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14Tv25fLSqTnZQiDzrYqKCSbdt6ep3upgqD2Se8QSRuQFynkSN): protocol: expected piece hash; context canceled; EOF Dec 29 03:37:46.168136090 (564h8m46.039219125s) [noise=true , address=64.49.13.26:4950 , piece_id=HQ6BBIFQAD5RRN7FRWGDFHAVVMSDUWJUGTX7LBTA2TQJ3PNWKUHA, seg_pos={0 0}, upload=494622]: Long tail cancellation due to optimal threshold Dec 29 03:37:46.168274995 (564h8m46.03935804s) [noise=true , address=77.90.31.12:4600 , piece_id=N4KXD5SYPYOKBXQRSL3UKNDDV7XB4KOKRYRYKGMMAHMAO6VVOEHQ, seg_pos={0 0}, upload=494622]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PKT2EcU6jAe6oUUdiTcwxGQsFa36uEmown77SemqQMM5Xw5nc): protocol: expected piece hash; context canceled; EOF Dec 29 03:37:46.168283491 (564h8m46.039366536s) [noise=true , address=77.90.31.12:4600 , piece_id=N4KXD5SYPYOKBXQRSL3UKNDDV7XB4KOKRYRYKGMMAHMAO6VVOEHQ, seg_pos={0 0}, upload=494622]: Long tail cancellation due to optimal threshold Dec 29 03:37:46.168394032 (564h8m46.039477067s) [noise=true , address=145.79.227.40:4068 , piece_id=4QOJWRVEE3ERM2AHP764XSTE7ZYHUUJ2ENBVXRS5UDIGKZOK7YKA, seg_pos={0 0}, upload=494622]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Lh9dYpHCxbs7gpZ4Xt7os5Gm5Nco6jrNW1V1FHddVUEMkFfPY): protocol: expected piece hash; context canceled; EOF Dec 29 03:37:46.168403810 (564h8m46.039486845s) [noise=true , address=145.79.227.40:4068 , piece_id=4QOJWRVEE3ERM2AHP764XSTE7ZYHUUJ2ENBVXRS5UDIGKZOK7YKA, seg_pos={0 0}, upload=494622]: Long tail cancellation due to optimal threshold Dec 29 03:37:46.168432725 (564h8m46.03951577s) [seg_pos={0 0}, upload=494622]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 03:37:46.168576960 (564h8m46.039659995s) [upload=494622]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 03:37:46.168584033 (564h8m46.039667068s) [upload=494622]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 03:37:46.466683658 (564h8m46.337766723s) [upload=494622]: Done uploading segments... Dec 29 03:37:46.466712052 (564h8m46.337795087s) [upload=494622]: Upload finished. err: <nil> Duration: 33.2172943s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [..................................XX................X......X] duration: 5.00121498s trace: https://jaeger.dev.storj.io/trace/3d6d508c8a6e1706 FAIL (2025-12-29 03:39:14 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001238365s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 29 out of 60 is failed (threshold=3) [XX..XX.....XX.X.....X.X..XXXX..XXX.XX...X.X.....XXX...XXXXXX] duration: 30.006625607s trace: https://jaeger.dev.storj.io/trace/2222aedb10f98fd1 FAIL (2025-12-29 03:39:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 29 03:38:33.809284564 (564h9m33.680367609s) [upload=494634]: Starting upload Dec 29 03:38:33.809362963 (564h9m33.680446008s) [upload=494634]: Done starting upload Dec 29 03:38:33.809670028 (564h9m33.680753073s) [upload=494634]: Uploading segments... Dec 29 03:38:33.809725383 (564h9m33.680808439s) [upload=494634]: Got next segment. Inline: false Dec 29 03:38:33.809763035 (564h9m33.68084608s) [upload=494634]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 03:38:33.809771421 (564h9m33.680854456s) [upload=494634]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 03:38:34.053876813 (564h9m33.924959858s) [seg_pos={0 0}, upload=494634]: Begin upload segment... Dec 29 03:38:34.053950804 (564h9m33.925033839s) [seg_pos={0 0}, upload=494634]: Done begin upload segment. Dec 29 03:38:34.054010989 (564h9m33.925094034s) [noise=true , address=77.90.31.12:4589 , piece_id=YWAHOSO2L2EXSQS4E4JXQTDD3TI7ZW2ZAP2YFCX4EMOVJ3RU3A2Q, seg_pos={0 0}, upload=494634]: Uploading piece... Dec 29 03:38:34.054038201 (564h9m33.925121236s) [noise=true , address=145.79.227.2:4113 , piece_id=YDNDGOEIDL4CHZHHW3AMIF7YEAE5VBHJPD7G7S7EYOG7PM3NUYMQ, seg_pos={0 0}, upload=494634]: Uploading piece... Dec 29 03:38:34.054747722 (564h9m33.925830767s) [noise=true , address=145.79.227.6:4259 , piece_id=KMS73EGS3X7ZGRFFWHWXWK3LVIAZFUXHXUSJYTJTLNK5M7GCKTDA, seg_pos={0 0}, upload=494634]: Uploading piece... Dec 29 03:38:34.055149867 (564h9m33.926232912s) [noise=true , address=145.79.227.15:4875 , piece_id=URICDEQAC3SXVFHZQTHNLKAWASD7YGPZ6JCGB7Z6SVLJD46CP22Q, seg_pos={0 0}, upload=494634]: Uploading piece... Dec 29 03:38:34.056308153 (564h9m33.927391188s) [noise=true , address=145.79.227.30:4761 , piece_id=I4OCBNGO3726QQZPOPJP4NVN3DOI4Q67IKZLCCCPNM3HIPAEAKNA, seg_pos={0 0}, upload=494634]: Uploading piece... Dec 29 03:38:34.056813445 (564h9m33.92789649s) [noise=true , address=77.90.31.15:4697 , piece_id=E5NB5UKU3HLATM7U64IRNAWY5IFYCGQIYPZHO3OCQCD2SZI647RQ, seg_pos={0 0}, upload=494634]: Uploading piece... Dec 29 03:38:34.058047615 (564h9m33.92913066s) [noise=true , address=145.79.227.26:4654 , piece_id=7L55HBYDNGUN5TBW7DKL4MKXBRPFLH4OXFMJ4OVALDXLI5CHRK4A, seg_pos={0 0}, upload=494634]: Uploading piece... Dec 29 03:38:34.058446965 (564h9m33.92953001s) [noise=true , address=77.90.31.19:4842 , piece_id=6GRC77URCWUVLLW7DMNF6XCLCAY5QCR7HSOBCBSMFL5JGT5QUNDQ, seg_pos={0 0}, upload=494634]: Uploading piece... Dec 29 03:38:34.059606042 (564h9m33.930689077s) [noise=true , address=145.79.227.47:4266 , piece_id=EN5UO2TJV623WLQGECGLYBXNOAK2GDCDMRTYQQAYKWC4YCEEXWSQ, seg_pos={0 0}, upload=494634]: Uploading piece... Dec 29 03:38:34.060050579 (564h9m33.931133634s) [noise=true , address=145.79.227.41:4087 , piece_id=TJUOPLUCCMOF7E7QLF6E2ICV5FO2WSY4CLOIZR5CCQMDEBXFHQVA, seg_pos={0 0}, upload=494634]: Uploading piece... Dec 29 03:38:34.061691734 (564h9m33.932774779s) [noise=true , address=77.90.31.14:4662 , piece_id=7NHO2J25S72AOOMG7PBRW4VA3JJPZSRPWY5BAAOF3P5H7GHDLIDQ, seg_pos={0 0}, upload=494634]: Uploading piece... Dec 29 03:38:34.063780040 (564h9m33.934863085s) [noise=true , address=45.91.116.9:4261 , piece_id=SXFY46DWYUGTEO764OWQVL4UWENQU4F2DT2XKQ5TE2G4VPKQUCMQ, seg_pos={0 0}, upload=494634]: Uploading piece... Dec 29 03:38:34.065939432 (564h9m33.937022477s) [noise=true , address=45.91.116.13:4155 , piece_id=GYDHPXFNUQ2DLWSEACXYPWTU35C7E5YOEK6ISLGFSQHOROWYFOTQ, seg_pos={0 0}, upload=494634]: Uploading piece... Dec 29 03:38:34.067095885 (564h9m33.93817892s) [noise=true , address=145.79.227.19:4077 , piece_id=UQWQJ3KBIW3QE2HCKKSTQIM4UIU2Q26JNUHO2XQJEJIJDDBEET6A, seg_pos={0 0}, upload=494634]: Uploading piece... Dec 29 03:38:34.070897824 (564h9m33.941980859s) [noise=true , address=145.79.227.11:4731 , piece_id=2LVLAHRAN64CH4UT7QMWZR5QEGC7CWNRPLFCSSIAUQ2BJVIRNAMA, seg_pos={0 0}, upload=494634]: Uploading piece... Dec 29 03:38:34.071753473 (564h9m33.942836518s) [noise=true , address=45.91.116.7:4012 , piece_id=UEEMQ4O7PS4P6G4NE6KFKQYZN7B3QBTYMHKGPGLKYJTGAVPEFHTQ, seg_pos={0 0}, upload=494634]: Uploading piece... Dec 29 03:38:35.749565138 (564h9m35.620648173s) [upload=494634]: Next returned nil segment Dec 29 03:38:35.749578173 (564h9m35.620661209s) [upload=494634]: Waiting for error group managing segments... Dec 29 03:38:36.048250248 (564h9m35.919333283s) [noise=true , address=145.79.227.41:4087 , piece_id=TJUOPLUCCMOF7E7QLF6E2ICV5FO2WSY4CLOIZR5CCQMDEBXFHQVA, seg_pos={0 0}, upload=494634]: Done uploading piece. err: <nil> Dec 29 03:38:36.050822795 (564h9m35.921905832s) [noise=true , address=145.79.227.19:4077 , piece_id=UQWQJ3KBIW3QE2HCKKSTQIM4UIU2Q26JNUHO2XQJEJIJDDBEET6A, seg_pos={0 0}, upload=494634]: Done uploading piece. err: <nil> Dec 29 03:38:36.051567534 (564h9m35.922650579s) [noise=true , address=145.79.227.11:4731 , piece_id=2LVLAHRAN64CH4UT7QMWZR5QEGC7CWNRPLFCSSIAUQ2BJVIRNAMA, seg_pos={0 0}, upload=494634]: Done uploading piece. err: <nil> Dec 29 03:38:36.052444304 (564h9m35.923527339s) [noise=true , address=145.79.227.15:4875 , piece_id=URICDEQAC3SXVFHZQTHNLKAWASD7YGPZ6JCGB7Z6SVLJD46CP22Q, seg_pos={0 0}, upload=494634]: Done uploading piece. err: <nil> Dec 29 03:38:36.058063754 (564h9m35.92914679s) [noise=true , address=145.79.227.2:4113 , piece_id=YDNDGOEIDL4CHZHHW3AMIF7YEAE5VBHJPD7G7S7EYOG7PM3NUYMQ, seg_pos={0 0}, upload=494634]: Done uploading piece. err: <nil> Dec 29 03:38:36.070523619 (564h9m35.941606654s) [noise=true , address=145.79.227.26:4654 , piece_id=7L55HBYDNGUN5TBW7DKL4MKXBRPFLH4OXFMJ4OVALDXLI5CHRK4A, seg_pos={0 0}, upload=494634]: Done uploading piece. err: <nil> Dec 29 03:38:36.085661433 (564h9m35.956744468s) [noise=true , address=145.79.227.47:4266 , piece_id=EN5UO2TJV623WLQGECGLYBXNOAK2GDCDMRTYQQAYKWC4YCEEXWSQ, seg_pos={0 0}, upload=494634]: Done uploading piece. err: <nil> Dec 29 03:38:36.100766075 (564h9m35.97184911s) [noise=true , address=145.79.227.6:4259 , piece_id=KMS73EGS3X7ZGRFFWHWXWK3LVIAZFUXHXUSJYTJTLNK5M7GCKTDA, seg_pos={0 0}, upload=494634]: Done uploading piece. err: <nil> Dec 29 03:38:36.243856400 (564h9m36.114939435s) [noise=true , address=77.90.31.15:4697 , piece_id=E5NB5UKU3HLATM7U64IRNAWY5IFYCGQIYPZHO3OCQCD2SZI647RQ, seg_pos={0 0}, upload=494634]: Done uploading piece. err: <nil> Dec 29 03:38:36.369044977 (564h9m36.240128022s) [noise=true , address=77.90.31.19:4842 , piece_id=6GRC77URCWUVLLW7DMNF6XCLCAY5QCR7HSOBCBSMFL5JGT5QUNDQ, seg_pos={0 0}, upload=494634]: Done uploading piece. err: <nil> Dec 29 03:38:36.429364744 (564h9m36.300447779s) [noise=true , address=77.90.31.12:4589 , piece_id=YWAHOSO2L2EXSQS4E4JXQTDD3TI7ZW2ZAP2YFCX4EMOVJ3RU3A2Q, seg_pos={0 0}, upload=494634]: Done uploading piece. err: <nil> Dec 29 03:38:36.438569582 (564h9m36.309652627s) [noise=true , address=77.90.31.14:4662 , piece_id=7NHO2J25S72AOOMG7PBRW4VA3JJPZSRPWY5BAAOF3P5H7GHDLIDQ, seg_pos={0 0}, upload=494634]: Done uploading piece. err: <nil> Dec 29 03:38:36.459456063 (564h9m36.330539098s) [noise=true , address=45.91.116.13:4155 , piece_id=GYDHPXFNUQ2DLWSEACXYPWTU35C7E5YOEK6ISLGFSQHOROWYFOTQ, seg_pos={0 0}, upload=494634]: Done uploading piece. err: <nil> Dec 29 03:38:36.459476832 (564h9m36.330559867s) [seg_pos={0 0}, upload=494634]: Segment reached optimal threshold of 13 pieces. Dec 29 03:38:36.459671734 (564h9m36.330754769s) [noise=true , address=45.91.116.9:4261 , piece_id=SXFY46DWYUGTEO764OWQVL4UWENQU4F2DT2XKQ5TE2G4VPKQUCMQ, seg_pos={0 0}, upload=494634]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: write tcp 188.166.225.170:58018->45.91.116.9:4261: use of closed network connection; write tcp 188.166.225.170:58018->45.91.116.9:4261: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58018->45.91.116.9:4261: use of closed network connection Dec 29 03:38:36.459679789 (564h9m36.330762824s) [noise=true , address=45.91.116.9:4261 , piece_id=SXFY46DWYUGTEO764OWQVL4UWENQU4F2DT2XKQ5TE2G4VPKQUCMQ, seg_pos={0 0}, upload=494634]: Long tail cancellation due to optimal threshold Dec 29 03:38:36.459692293 (564h9m36.330775328s) [noise=true , address=45.91.116.7:4012 , piece_id=UEEMQ4O7PS4P6G4NE6KFKQYZN7B3QBTYMHKGPGLKYJTGAVPEFHTQ, seg_pos={0 0}, upload=494634]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t): protocol: expected piece hash; context canceled; EOF Dec 29 03:38:36.459700709 (564h9m36.330783744s) [noise=true , address=45.91.116.7:4012 , piece_id=UEEMQ4O7PS4P6G4NE6KFKQYZN7B3QBTYMHKGPGLKYJTGAVPEFHTQ, seg_pos={0 0}, upload=494634]: Long tail cancellation due to optimal threshold Dec 29 03:38:36.459874279 (564h9m36.330957314s) [noise=true , address=145.79.227.30:4761 , piece_id=I4OCBNGO3726QQZPOPJP4NVN3DOI4Q67IKZLCCCPNM3HIPAEAKNA, seg_pos={0 0}, upload=494634]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF Dec 29 03:38:36.459888777 (564h9m36.330971812s) [noise=true , address=145.79.227.30:4761 , piece_id=I4OCBNGO3726QQZPOPJP4NVN3DOI4Q67IKZLCCCPNM3HIPAEAKNA, seg_pos={0 0}, upload=494634]: Long tail cancellation due to optimal threshold Dec 29 03:38:36.459915488 (564h9m36.330998523s) [seg_pos={0 0}, upload=494634]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 03:38:36.460008996 (564h9m36.331092031s) [upload=494634]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 03:38:36.460014246 (564h9m36.331097281s) [upload=494634]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 03:38:36.749698446 (564h9m36.620781491s) [upload=494634]: Done uploading segments... Dec 29 03:38:36.749721269 (564h9m36.620804314s) [upload=494634]: Upload finished. err: <nil> Duration: 30.006652468s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [...................................XX................X......] duration: 3.516119895s trace: https://jaeger.dev.storj.io/trace/611c7ac3f284e077 FAIL (2025-12-29 03:38:43 UTC) orbiter warning: context deadline exceeded Duration: 3.516147709s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......................................X...................X] duration: 1m0.002790925s trace: https://jaeger.dev.storj.io/trace/2408171d336afafd FAIL (2025-12-29 03:38:42 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12Tz3HEHACmHU9Gp2W2vHVuF5PXNmdsa5N8nYhwt52HCTjggqRD, Piece ID: S3MEER2QCVCORYQNYNY2S22TC2H5HHKZW7VQJ54VL754UZTD42JQ) context deadline exceeded; piecestore: (Node ID: 1ve7KKYDbNo8Ma7wGVBH5x9z7zbPtPAPztmuQ9r6ijPpVgh5NX, Piece ID: S7T55BAWP53YJVHPW2GPLUM2VB3NBHZ2YD4UWBKKWCFHDAXAW3KQ) context deadline exceeded; piecestore: (Node ID: 1WVHPacX35JJ8PFzCs3UtGwsR4yVHXKmiqfGedK9KfXbHNJtmu, Piece ID: NYWKOTPLBZQIHGSQFMTT2QE6OQX7H5OKFVWJK3I65CWGGDSDDBMA) context deadline exceeded ======LOGS====== upload=494622]: Uploading piece... Dec 29 03:37:43.315384541 (564h8m43.186467596s) [noise=true , address=145.79.227.11:4762 , piece_id=2GQ5ASSBMUC52NCYRLBF3EOKYEA3G6DTIA6K56DE3OUQ4F5OLX2Q, seg_pos={0 0}, upload=494622]: Uploading piece... Dec 29 03:37:43.315899872 (564h8m43.186982907s) [noise=true , address=77.90.31.13:4649 , piece_id=S7T55BAWP53YJVHPW2GPLUM2VB3NBHZ2YD4UWBKKWCFHDAXAW3KQ, seg_pos={0 0}, upload=494622]: Uploading piece... Dec 29 03:37:43.316395165 (564h8m43.1874782s) [noise=true , address=77.90.31.18:4834 , piece_id=FQS6BFFJTORJPKJCINDZKXTY6GFRQTR3FZPWAEG7F2DNROUEPGJA, seg_pos={0 0}, upload=494622]: Uploading piece... Dec 29 03:37:43.316885961 (564h8m43.187969006s) [noise=true , address=193.149.164.143:4695 , piece_id=YF4EWRMS3GWEYPSA67GLG6EA5H7GZ55GTA3ISIEJ7RFEG2NDTLEQ, seg_pos={0 0}, upload=494622]: Uploading piece... Dec 29 03:37:43.317355795 (564h8m43.18843883s) [noise=true , address=145.79.227.7:4297 , piece_id=NNJ2KDDLNZHT4SGOUXBWBYD5OGW6MTGCAWZ63D5LH7BGM6KOJGDA, seg_pos={0 0}, upload=494622]: Uploading piece... Dec 29 03:37:43.317364631 (564h8m43.188447666s) [noise=true , address=145.79.227.33:4886 , piece_id=DB64UXSDC47SRTQX7PJGE66RCL7XFW3TGMCU3JOWVEJVGVHLYGWQ, seg_pos={0 0}, upload=494622]: Uploading piece... Dec 29 03:37:43.317689451 (564h8m43.188772486s) [noise=true , address=64.49.13.26:4950 , piece_id=HQ6BBIFQAD5RRN7FRWGDFHAVVMSDUWJUGTX7LBTA2TQJ3PNWKUHA, seg_pos={0 0}, upload=494622]: Uploading piece... Dec 29 03:37:43.317877448 (564h8m43.188960493s) [noise=true , address=45.91.116.10:4096 , piece_id=VIQK6EV67DONFZY77BOTWHY7POJEOE7FDHB6TKQMHVSHJJM35V7Q, seg_pos={0 0}, upload=494622]: Uploading piece... Dec 29 03:37:43.318315813 (564h8m43.189398858s) [noise=true , address=145.79.227.9:4388 , piece_id=DQU5TSAWTEEIBT272I76ZMGXD6FJT2QMXQISRDMWG5NOPORLBLLA, seg_pos={0 0}, upload=494622]: Uploading piece... Dec 29 03:37:43.318722317 (564h8m43.189805352s) [noise=true , address=77.90.31.12:4600 , piece_id=N4KXD5SYPYOKBXQRSL3UKNDDV7XB4KOKRYRYKGMMAHMAO6VVOEHQ, seg_pos={0 0}, upload=494622]: Uploading piece... Dec 29 03:37:43.319195248 (564h8m43.190278293s) [noise=true , address=45.91.116.8:4050 , piece_id=NYWKOTPLBZQIHGSQFMTT2QE6OQX7H5OKFVWJK3I65CWGGDSDDBMA, seg_pos={0 0}, upload=494622]: Uploading piece... Dec 29 03:37:43.319666174 (564h8m43.190749219s) [noise=true , address=77.90.31.21:4899 , piece_id=4MZKL4KVJF4DYGVPYGVOUHD6TYIDCRER33YPW6RG7KT6B6IXQEOA, seg_pos={0 0}, upload=494622]: Uploading piece... Dec 29 03:37:43.321557184 (564h8m43.192640229s) [noise=true , address=145.79.227.34:4909 , piece_id=UCGJPN357USFHIQ7II5GLRKNDE4H4HUVDQ7CB5R27C3VGRP2R3IA, seg_pos={0 0}, upload=494622]: Uploading piece... Dec 29 03:37:43.322231088 (564h8m43.193314133s) [noise=true , address=45.91.116.11:4139 , piece_id=S3MEER2QCVCORYQNYNY2S22TC2H5HHKZW7VQJ54VL754UZTD42JQ, seg_pos={0 0}, upload=494622]: Uploading piece... Dec 29 03:37:43.322851720 (564h8m43.193934755s) [noise=true , address=145.79.227.51:4444 , piece_id=EWEXXYYWF43J7FG6NPSJ2PICHTRPY533L4DQQX2CSW5IMJ6CB2PQ, seg_pos={0 0}, upload=494622]: Uploading piece... Dec 29 03:37:43.323435972 (564h8m43.194519007s) [noise=true , address=145.79.227.40:4068 , piece_id=4QOJWRVEE3ERM2AHP764XSTE7ZYHUUJ2ENBVXRS5UDIGKZOK7YKA, seg_pos={0 0}, upload=494622]: Uploading piece... Dec 29 03:37:43.323901118 (564h8m43.194984153s) [noise=true , address=145.79.227.30:4790 , piece_id=2TSIHAVWOZPOVKSTG2ZXBBEFEMMMRLMOWHM47TDANARFPDJM3NPQ, seg_pos={0 0}, upload=494622]: Uploading piece... Dec 29 03:37:44.620573279 (564h8m44.491656314s) [upload=494622]: Next returned nil segment Dec 29 03:37:44.620593617 (564h8m44.491676652s) [upload=494622]: Waiting for error group managing segments... Dec 29 03:37:45.178264102 (564h8m45.049347137s) [noise=true , address=145.79.227.11:4762 , piece_id=2GQ5ASSBMUC52NCYRLBF3EOKYEA3G6DTIA6K56DE3OUQ4F5OLX2Q, seg_pos={0 0}, upload=494622]: Done uploading piece. err: <nil> Dec 29 03:37:45.190189951 (564h8m45.061272986s) [noise=true , address=145.79.227.51:4444 , piece_id=EWEXXYYWF43J7FG6NPSJ2PICHTRPY533L4DQQX2CSW5IMJ6CB2PQ, seg_pos={0 0}, upload=494622]: Done uploading piece. err: <nil> Dec 29 03:37:45.199163818 (564h8m45.070246853s) [noise=true , address=145.79.227.7:4297 , piece_id=NNJ2KDDLNZHT4SGOUXBWBYD5OGW6MTGCAWZ63D5LH7BGM6KOJGDA, seg_pos={0 0}, upload=494622]: Done uploading piece. err: <nil> Dec 29 03:37:45.448727893 (564h8m45.319810928s) [noise=true , address=45.91.116.11:4139 , piece_id=S3MEER2QCVCORYQNYNY2S22TC2H5HHKZW7VQJ54VL754UZTD42JQ, seg_pos={0 0}, upload=494622]: Done uploading piece. err: <nil> Dec 29 03:37:45.459119140 (564h8m45.330202175s) [noise=true , address=77.90.31.13:4649 , piece_id=S7T55BAWP53YJVHPW2GPLUM2VB3NBHZ2YD4UWBKKWCFHDAXAW3KQ, seg_pos={0 0}, upload=494622]: Done uploading piece. err: <nil> Dec 29 03:37:45.471609714 (564h8m45.342692759s) [noise=true , address=45.91.116.8:4050 , piece_id=NYWKOTPLBZQIHGSQFMTT2QE6OQX7H5OKFVWJK3I65CWGGDSDDBMA, seg_pos={0 0}, upload=494622]: Done uploading piece. err: <nil> Dec 29 03:37:45.489098314 (564h8m45.36018136s) [noise=true , address=77.90.31.21:4899 , piece_id=4MZKL4KVJF4DYGVPYGVOUHD6TYIDCRER33YPW6RG7KT6B6IXQEOA, seg_pos={0 0}, upload=494622]: Done uploading piece. err: <nil> Dec 29 03:37:45.489726060 (564h8m45.360809105s) [noise=true , address=77.90.31.11:4574 , piece_id=3TP2ME5NTXVK45B34W5U4DESLFJLOJEPRQOJLEEPN2PIMYS2IDWQ, seg_pos={0 0}, upload=494622]: Done uploading piece. err: <nil> Dec 29 03:37:45.779669865 (564h8m45.65075291s) [noise=true , address=145.79.227.9:4388 , piece_id=DQU5TSAWTEEIBT272I76ZMGXD6FJT2QMXQISRDMWG5NOPORLBLLA, seg_pos={0 0}, upload=494622]: Done uploading piece. err: <nil> Dec 29 03:37:45.817564973 (564h8m45.688648008s) [noise=true , address=77.90.31.18:4834 , piece_id=FQS6BFFJTORJPKJCINDZKXTY6GFRQTR3FZPWAEG7F2DNROUEPGJA, seg_pos={0 0}, upload=494622]: Done uploading piece. err: <nil> Dec 29 03:37:46.001090802 (564h8m45.872173857s) [noise=true , address=193.149.164.143:4695 , piece_id=YF4EWRMS3GWEYPSA67GLG6EA5H7GZ55GTA3ISIEJ7RFEG2NDTLEQ, seg_pos={0 0}, upload=494622]: Done uploading piece. err: <nil> Dec 29 03:37:46.129753764 (564h8m46.000836819s) [noise=true , address=145.79.227.33:4886 , piece_id=DB64UXSDC47SRTQX7PJGE66RCL7XFW3TGMCU3JOWVEJVGVHLYGWQ, seg_pos={0 0}, upload=494622]: Done uploading piece. err: <nil> Dec 29 03:37:46.163280064 (564h8m46.034363109s) [noise=true , address=145.79.227.34:4909 , piece_id=UCGJPN357USFHIQ7II5GLRKNDE4H4HUVDQ7CB5R27C3VGRP2R3IA, seg_pos={0 0}, upload=494622]: Done uploading piece. err: <nil> Dec 29 03:37:46.167742170 (564h8m46.038825205s) [noise=true , address=145.79.227.30:4790 , piece_id=2TSIHAVWOZPOVKSTG2ZXBBEFEMMMRLMOWHM47TDANARFPDJM3NPQ, seg_pos={0 0}, upload=494622]: Done uploading piece. err: <nil> Dec 29 03:37:46.167764403 (564h8m46.038847448s) [seg_pos={0 0}, upload=494622]: Segment reached optimal threshold of 14 pieces. Dec 29 03:37:46.167996534 (564h8m46.039079569s) [noise=true , address=45.91.116.10:4096 , piece_id=VIQK6EV67DONFZY77BOTWHY7POJEOE7FDHB6TKQMHVSHJJM35V7Q, seg_pos={0 0}, upload=494622]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12MZX1GwZGrEi2q44dPiYrwMoum5A2caZtXTS3qiLNcoyHhon7p): protocol: expected piece hash; context canceled; EOF Dec 29 03:37:46.168005812 (564h8m46.039088857s) [noise=true , address=45.91.116.10:4096 , piece_id=VIQK6EV67DONFZY77BOTWHY7POJEOE7FDHB6TKQMHVSHJJM35V7Q, seg_pos={0 0}, upload=494622]: Long tail cancellation due to optimal threshold Dec 29 03:37:46.168120160 (564h8m46.039203195s) [noise=true , address=64.49.13.26:4950 , piece_id=HQ6BBIFQAD5RRN7FRWGDFHAVVMSDUWJUGTX7LBTA2TQJ3PNWKUHA, seg_pos={0 0}, upload=494622]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14Tv25fLSqTnZQiDzrYqKCSbdt6ep3upgqD2Se8QSRuQFynkSN): protocol: expected piece hash; context canceled; EOF Dec 29 03:37:46.168136090 (564h8m46.039219125s) [noise=true , address=64.49.13.26:4950 , piece_id=HQ6BBIFQAD5RRN7FRWGDFHAVVMSDUWJUGTX7LBTA2TQJ3PNWKUHA, seg_pos={0 0}, upload=494622]: Long tail cancellation due to optimal threshold Dec 29 03:37:46.168274995 (564h8m46.03935804s) [noise=true , address=77.90.31.12:4600 , piece_id=N4KXD5SYPYOKBXQRSL3UKNDDV7XB4KOKRYRYKGMMAHMAO6VVOEHQ, seg_pos={0 0}, upload=494622]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PKT2EcU6jAe6oUUdiTcwxGQsFa36uEmown77SemqQMM5Xw5nc): protocol: expected piece hash; context canceled; EOF Dec 29 03:37:46.168283491 (564h8m46.039366536s) [noise=true , address=77.90.31.12:4600 , piece_id=N4KXD5SYPYOKBXQRSL3UKNDDV7XB4KOKRYRYKGMMAHMAO6VVOEHQ, seg_pos={0 0}, upload=494622]: Long tail cancellation due to optimal threshold Dec 29 03:37:46.168394032 (564h8m46.039477067s) [noise=true , address=145.79.227.40:4068 , piece_id=4QOJWRVEE3ERM2AHP764XSTE7ZYHUUJ2ENBVXRS5UDIGKZOK7YKA, seg_pos={0 0}, upload=494622]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Lh9dYpHCxbs7gpZ4Xt7os5Gm5Nco6jrNW1V1FHddVUEMkFfPY): protocol: expected piece hash; context canceled; EOF Dec 29 03:37:46.168403810 (564h8m46.039486845s) [noise=true , address=145.79.227.40:4068 , piece_id=4QOJWRVEE3ERM2AHP764XSTE7ZYHUUJ2ENBVXRS5UDIGKZOK7YKA, seg_pos={0 0}, upload=494622]: Long tail cancellation due to optimal threshold Dec 29 03:37:46.168432725 (564h8m46.03951577s) [seg_pos={0 0}, upload=494622]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 03:37:46.168576960 (564h8m46.039659995s) [upload=494622]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 03:37:46.168584033 (564h8m46.039667068s) [upload=494622]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 03:37:46.466683658 (564h8m46.337766723s) [upload=494622]: Done uploading segments... Dec 29 03:37:46.466712052 (564h8m46.337795087s) [upload=494622]: Upload finished. err: <nil> Duration: 1m0.002812336s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 28 out of 60 is failed (threshold=3) [.XX..XX.....XX.X.....X.X..XXXX..XXX.XX...X.X.....XXX...XXXXX] duration: 30.003831039s trace: https://jaeger.dev.storj.io/trace/6218c6328ad159ba FAIL (2025-12-29 03:38:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, Piece ID: K6KZLUSGA7YEFHFIMUBIFKW2T5NNQUZIXLPZYVHH5ODO6D5VWZRA) context deadline exceeded; piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: CQ3PTFWEUCWIOTTSRE6VFWD74NDH46YR4SL6TEQD4NOLSBQGENLA) context deadline exceeded; piecestore: (Node ID: 121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp, Piece ID: CH642NFMOMRBG3SAOSRDZ5H74JHUZN64JPYGCLBQAC3QIJQR5XLA) context deadline exceeded ======LOGS====== Dec 29 03:37:51.775433780 (564h8m51.646516815s) [upload=494624]: Starting upload Dec 29 03:37:51.775494756 (564h8m51.646577801s) [upload=494624]: Done starting upload Dec 29 03:37:51.775678457 (564h8m51.646761502s) [upload=494624]: Uploading segments... Dec 29 03:37:51.775778055 (564h8m51.6468611s) [upload=494624]: Got next segment. Inline: false Dec 29 03:37:51.775834684 (564h8m51.646917719s) [upload=494624]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 03:37:51.775841116 (564h8m51.646924171s) [upload=494624]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 03:37:52.030321923 (564h8m51.901404988s) [seg_pos={0 0}, upload=494624]: Begin upload segment... Dec 29 03:37:52.030376186 (564h8m51.901459231s) [seg_pos={0 0}, upload=494624]: Done begin upload segment. Dec 29 03:37:52.030404280 (564h8m51.901487315s) [noise=true , address=145.79.227.11:4731 , piece_id=6IXJSF6ENUMTTDUJCT4AK5SE26YV22KC6DFSDKHKEI2FEIANJJMA, seg_pos={0 0}, upload=494624]: Uploading piece... Dec 29 03:37:52.030434828 (564h8m51.901517863s) [noise=true , address=145.79.227.16:4909 , piece_id=36MFSEXLM3GDHKUYY2B4WR4YEB5TDNI34HOWHGX64SNIJQ3O46EQ, seg_pos={0 0}, upload=494624]: Uploading piece... Dec 29 03:37:52.030950499 (564h8m51.902033545s) [noise=true , address=45.91.116.10:4082 , piece_id=TUBB4WCM5V7R53AHAZR4U4YRF2T7F47VY3TI6JSNIPIIMA445N5A, seg_pos={0 0}, upload=494624]: Uploading piece... Dec 29 03:37:52.031224902 (564h8m51.902307947s) [noise=true , address=145.79.227.13:4803 , piece_id=HJBBYFPW3OVFJEFPAOFGXZLW6DQCRCK4DEZQNXLCGKFLPY5RCXSQ, seg_pos={0 0}, upload=494624]: Uploading piece... Dec 29 03:37:52.031261311 (564h8m51.902344346s) [noise=true , address=77.90.31.14:4661 , piece_id=K6KZLUSGA7YEFHFIMUBIFKW2T5NNQUZIXLPZYVHH5ODO6D5VWZRA, seg_pos={0 0}, upload=494624]: Uploading piece... Dec 29 03:37:52.031576151 (564h8m51.902659186s) [noise=true , address=45.91.116.5:4299 , piece_id=CQ3PTFWEUCWIOTTSRE6VFWD74NDH46YR4SL6TEQD4NOLSBQGENLA, seg_pos={0 0}, upload=494624]: Uploading piece... Dec 29 03:37:52.031799677 (564h8m51.902882712s) [noise=true , address=45.91.116.6:4976 , piece_id=CH642NFMOMRBG3SAOSRDZ5H74JHUZN64JPYGCLBQAC3QIJQR5XLA, seg_pos={0 0}, upload=494624]: Uploading piece... Dec 29 03:37:52.031865042 (564h8m51.902948087s) [noise=true , address=145.79.227.51:4410 , piece_id=ZZWLHJEDPV5KVKPRNJYKFW4TBRD7INYTA4JU24XTMY3RFBXGCQCQ, seg_pos={0 0}, upload=494624]: Uploading piece... Dec 29 03:37:52.032140787 (564h8m51.903223822s) [noise=true , address=145.79.227.29:4942 , piece_id=MFDVBCYDNCOIZSTBONVZFB7KQ56TCNEQVKGGWPMEVXAP2TEULCXA, seg_pos={0 0}, upload=494624]: Uploading piece... Dec 29 03:37:52.032259763 (564h8m51.903342798s) [noise=true , address=145.79.227.17:4972 , piece_id=WUDUWG4OAZULVRWEWKAZS62PDYHZHU22BHN3SZPJQRTPUG6QKWUQ, seg_pos={0 0}, upload=494624]: Uploading piece... Dec 29 03:37:52.032531400 (564h8m51.903614435s) [noise=true , address=145.79.227.48:4302 , piece_id=NZSKSFIBXVZK5G5QYRRVFMHOEVVQ3ILLLJL5NRIOUVUITSMRRDWA, seg_pos={0 0}, upload=494624]: Uploading piece... Dec 29 03:37:52.032630509 (564h8m51.903713544s) [noise=true , address=145.79.227.15:4873 , piece_id=3GPQIKFAUJZJJPCHF7CFZ5RNNSSQUCT4KM3RRZCFRYWJU63OH6RQ, seg_pos={0 0}, upload=494624]: Uploading piece... Dec 29 03:37:52.032851760 (564h8m51.903934805s) [noise=true , address=45.91.116.14:4190 , piece_id=7A3JVA57USX5ABFGAM3NBQLMLVVHG6QQSIFLZ73TURZQ7KHCCYIA, seg_pos={0 0}, upload=494624]: Uploading piece... Dec 29 03:37:52.032919900 (564h8m51.904002935s) [noise=true , address=145.79.227.44:4194 , piece_id=YIDX63DSTZQ3SLFP46APTHJGJ45P4PBEJL7AROOI7QK3TL5CPEIQ, seg_pos={0 0}, upload=494624]: Uploading piece... Dec 29 03:37:52.033237264 (564h8m51.904320309s) [noise=true , address=145.79.227.26:4653 , piece_id=MIY4VQOMEJES44DKGJDNTWV6VG6O5XV625AZ5XTWTHKQOKVBBY2Q, seg_pos={0 0}, upload=494624]: Uploading piece... Dec 29 03:37:52.033674457 (564h8m51.904757492s) [noise=true , address=145.79.227.21:4473 , piece_id=IGIZ6DRB7LFMXG2MYAJUMZBMGZ6BRABT6637WZM6KXFWGUS42QIQ, seg_pos={0 0}, upload=494624]: Uploading piece... Dec 29 03:37:53.730319692 (564h8m53.601402728s) [upload=494624]: Next returned nil segment Dec 29 03:37:53.730332837 (564h8m53.601415872s) [upload=494624]: Waiting for error group managing segments... Dec 29 03:37:54.002225981 (564h8m53.873309046s) [noise=true , address=145.79.227.15:4873 , piece_id=3GPQIKFAUJZJJPCHF7CFZ5RNNSSQUCT4KM3RRZCFRYWJU63OH6RQ, seg_pos={0 0}, upload=494624]: Done uploading piece. err: <nil> Dec 29 03:37:54.002763665 (564h8m53.873846721s) [noise=true , address=145.79.227.11:4731 , piece_id=6IXJSF6ENUMTTDUJCT4AK5SE26YV22KC6DFSDKHKEI2FEIANJJMA, seg_pos={0 0}, upload=494624]: Done uploading piece. err: <nil> Dec 29 03:37:54.014420301 (564h8m53.885503336s) [noise=true , address=145.79.227.48:4302 , piece_id=NZSKSFIBXVZK5G5QYRRVFMHOEVVQ3ILLLJL5NRIOUVUITSMRRDWA, seg_pos={0 0}, upload=494624]: Done uploading piece. err: <nil> Dec 29 03:37:54.023837253 (564h8m53.894920298s) [noise=true , address=145.79.227.17:4972 , piece_id=WUDUWG4OAZULVRWEWKAZS62PDYHZHU22BHN3SZPJQRTPUG6QKWUQ, seg_pos={0 0}, upload=494624]: Done uploading piece. err: <nil> Dec 29 03:37:54.032246635 (564h8m53.90332967s) [noise=true , address=145.79.227.26:4653 , piece_id=MIY4VQOMEJES44DKGJDNTWV6VG6O5XV625AZ5XTWTHKQOKVBBY2Q, seg_pos={0 0}, upload=494624]: Done uploading piece. err: <nil> Dec 29 03:37:54.035564372 (564h8m53.906647417s) [noise=true , address=145.79.227.16:4909 , piece_id=36MFSEXLM3GDHKUYY2B4WR4YEB5TDNI34HOWHGX64SNIJQ3O46EQ, seg_pos={0 0}, upload=494624]: Done uploading piece. err: <nil> Dec 29 03:37:54.036209321 (564h8m53.907292366s) [noise=true , address=145.79.227.51:4410 , piece_id=ZZWLHJEDPV5KVKPRNJYKFW4TBRD7INYTA4JU24XTMY3RFBXGCQCQ, seg_pos={0 0}, upload=494624]: Done uploading piece. err: <nil> Dec 29 03:37:54.037539273 (564h8m53.908622308s) [noise=true , address=145.79.227.13:4803 , piece_id=HJBBYFPW3OVFJEFPAOFGXZLW6DQCRCK4DEZQNXLCGKFLPY5RCXSQ, seg_pos={0 0}, upload=494624]: Done uploading piece. err: <nil> Dec 29 03:37:54.038018055 (564h8m53.90910109s) [noise=true , address=145.79.227.21:4473 , piece_id=IGIZ6DRB7LFMXG2MYAJUMZBMGZ6BRABT6637WZM6KXFWGUS42QIQ, seg_pos={0 0}, upload=494624]: Done uploading piece. err: <nil> Dec 29 03:37:54.278062964 (564h8m54.149146009s) [noise=true , address=45.91.116.5:4299 , piece_id=CQ3PTFWEUCWIOTTSRE6VFWD74NDH46YR4SL6TEQD4NOLSBQGENLA, seg_pos={0 0}, upload=494624]: Done uploading piece. err: <nil> Dec 29 03:37:54.280496568 (564h8m54.151579614s) [noise=true , address=45.91.116.10:4082 , piece_id=TUBB4WCM5V7R53AHAZR4U4YRF2T7F47VY3TI6JSNIPIIMA445N5A, seg_pos={0 0}, upload=494624]: Done uploading piece. err: <nil> Dec 29 03:37:54.335981459 (564h8m54.207064504s) [noise=true , address=45.91.116.6:4976 , piece_id=CH642NFMOMRBG3SAOSRDZ5H74JHUZN64JPYGCLBQAC3QIJQR5XLA, seg_pos={0 0}, upload=494624]: Done uploading piece. err: <nil> Dec 29 03:37:54.924013146 (564h8m54.795096181s) [noise=true , address=77.90.31.14:4661 , piece_id=K6KZLUSGA7YEFHFIMUBIFKW2T5NNQUZIXLPZYVHH5ODO6D5VWZRA, seg_pos={0 0}, upload=494624]: Done uploading piece. err: <nil> Dec 29 03:37:54.924031430 (564h8m54.795114475s) [seg_pos={0 0}, upload=494624]: Segment reached optimal threshold of 13 pieces. Dec 29 03:37:54.924187348 (564h8m54.795270383s) [noise=true , address=45.91.116.14:4190 , piece_id=7A3JVA57USX5ABFGAM3NBQLMLVVHG6QQSIFLZ73TURZQ7KHCCYIA, seg_pos={0 0}, upload=494624]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: write tcp 188.166.225.170:40136->45.91.116.14:4190: use of closed network connection; write tcp 188.166.225.170:40136->45.91.116.14:4190: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40136->45.91.116.14:4190: use of closed network connection Dec 29 03:37:54.924192798 (564h8m54.795275833s) [noise=true , address=45.91.116.14:4190 , piece_id=7A3JVA57USX5ABFGAM3NBQLMLVVHG6QQSIFLZ73TURZQ7KHCCYIA, seg_pos={0 0}, upload=494624]: Long tail cancellation due to optimal threshold Dec 29 03:37:54.924369064 (564h8m54.795452099s) [noise=true , address=145.79.227.29:4942 , piece_id=MFDVBCYDNCOIZSTBONVZFB7KQ56TCNEQVKGGWPMEVXAP2TEULCXA, seg_pos={0 0}, upload=494624]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17o881pUMuG7K2dv83k7RkrJCrmM7kgUiWesgam4GEw21KVUn7): protocol: expected piece hash; context canceled; EOF Dec 29 03:37:54.924375045 (564h8m54.79545808s) [noise=true , address=145.79.227.29:4942 , piece_id=MFDVBCYDNCOIZSTBONVZFB7KQ56TCNEQVKGGWPMEVXAP2TEULCXA, seg_pos={0 0}, upload=494624]: Long tail cancellation due to optimal threshold Dec 29 03:37:54.924453515 (564h8m54.79553655s) [noise=true , address=145.79.227.44:4194 , piece_id=YIDX63DSTZQ3SLFP46APTHJGJ45P4PBEJL7AROOI7QK3TL5CPEIQ, seg_pos={0 0}, upload=494624]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh): protocol: expected piece hash; context canceled; EOF Dec 29 03:37:54.924474374 (564h8m54.795557409s) [noise=true , address=145.79.227.44:4194 , piece_id=YIDX63DSTZQ3SLFP46APTHJGJ45P4PBEJL7AROOI7QK3TL5CPEIQ, seg_pos={0 0}, upload=494624]: Long tail cancellation due to optimal threshold Dec 29 03:37:54.924508249 (564h8m54.795591284s) [seg_pos={0 0}, upload=494624]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 03:37:54.924631043 (564h8m54.795714078s) [upload=494624]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 03:37:54.924636243 (564h8m54.795719278s) [upload=494624]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 03:37:55.216746082 (564h8m55.087829137s) [upload=494624]: Done uploading segments... Dec 29 03:37:55.216827677 (564h8m55.087910712s) [upload=494624]: Upload finished. err: <nil> Duration: 30.003865835s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [....................................XX................X.....] duration: 4.625654297s trace: https://jaeger.dev.storj.io/trace/611c7ac3f284e077 FAIL (2025-12-29 03:37:59 UTC) orbiter warning: context deadline exceeded Duration: 4.625673684s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 56.232845346s trace: https://jaeger.dev.storj.io/trace/1584175ba8dba7fe FAIL (2025-12-29 03:37:30 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ece... Dec 29 03:20:05.592697598 (563h51m5.463780643s) [noise=true , address=77.90.31.16:4739 , piece_id=5I3LCPAI2IF5Z5MZWQNKDM7LKBYEYIPE3EBUAD6DOLBPW34FDNPA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.593249339 (563h51m5.464332384s) [noise=true , address=77.90.31.12:4605 , piece_id=CEYXIJYUE4MGBAOGMKOWIQZBRFRC23FFUZCP5AW7N4MCT76B5DWQ, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.593687443 (563h51m5.464770488s) [noise=true , address=145.79.227.49:4355 , piece_id=ZBF32JIHB33TYP2KB427V7OQRRNXP2ZOTLZNFRQ5XUH7HGPG2RTA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.594173598 (563h51m5.465256633s) [noise=true , address=145.79.227.2:4142 , piece_id=GS7XTVLRWWXMZKGYSVDGKAGR6QX3OJUTK4TNNVPPA26XS5KSHKBQ, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.594639957 (563h51m5.465723002s) [noise=true , address=145.79.227.9:4382 , piece_id=WQDVG2KXSEDZHJJOT2ZFSAPQXVLJZXMO5HQXXEU6XJ7U7CAQWZVQ, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.595331994 (563h51m5.466415039s) [noise=true , address=145.79.227.43:4170 , piece_id=F7H3IF2SNNGTIV2CFKXB3GIA6TGRLPSOLA6TPBUW5FQVWULEK4GQ, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.595834151 (563h51m5.466917206s) [noise=true , address=145.79.227.12:4778 , piece_id=KHBVVV3WSILPU4CQK4HAUC37D75DUWGOJWZFY53SBDLVY3F3EBAA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.596329844 (563h51m5.467412899s) [noise=true , address=145.79.227.50:4391 , piece_id=GX6GJKCSCDKGMYRAPNP4Z347JJMKPWFLDT2QGOSSZGAYKNZNTDHA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.599128643 (563h51m5.470211688s) [noise=true , address=45.91.116.13:4177 , piece_id=YIJBXUI5PGKEJOY3USREMMV35V2VMXXFILDOUNJPXMIHVD5PZIUA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.599196953 (563h51m5.470279988s) [noise=true , address=145.79.227.35:4493 , piece_id=A2ZOH3MMXV5S7JPOWOCQZMYQEMF3SJHTYIIOB2656AVAXP24UAYA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.601094367 (563h51m5.472177412s) [noise=true , address=77.90.31.18:4838 , piece_id=TP42RP5AUZSP6LNKK7ON4FMDFVF6GRX3FLRYWBS56SCKFAZNPQSA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.602868896 (563h51m5.473951941s) [noise=true , address=77.90.31.21:4909 , piece_id=KKTXOQ647SWOMJ7J4DC6CMHBSI65JJD2C5IMZZWI3ZSQI7SXLY2A, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.604661328 (563h51m5.475744374s) [noise=true , address=77.90.31.13:4644 , piece_id=VJZEITJXVPJHT6FPHXEFDMWDGR4XQNAZMDGP7BOKEQEAKKN5LKJQ, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.606462138 (563h51m5.477545183s) [noise=true , address=145.79.227.35:4507 , piece_id=SNS3BVKFWUKQYMNZCWMOHH563GXNM5IIB2QMLGVCCMIXHEV7MJOA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.609580124 (563h51m5.480663169s) [noise=true , address=64.49.13.140:4515 , piece_id=BB4RND6DQPMXBJHNTIFIQWVZOFZFHIFB6KBYEUOKOLJFDCWRIMJA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.611644807 (563h51m5.482727852s) [noise=true , address=77.90.31.11:4588 , piece_id=XU7QUQZHRH27X64OR4NXJERPPIHCPBEGL7POHUL5Z32LTFMI5MKQ, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:06.901416710 (563h51m6.772499755s) [upload=494362]: Next returned nil segment Dec 29 03:20:06.901434123 (563h51m6.772517168s) [upload=494362]: Waiting for error group managing segments... Dec 29 03:20:07.237383793 (563h51m7.108466848s) [noise=true , address=64.49.13.140:4515 , piece_id=BB4RND6DQPMXBJHNTIFIQWVZOFZFHIFB6KBYEUOKOLJFDCWRIMJA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.439111607 (563h51m7.310194652s) [noise=true , address=145.79.227.9:4382 , piece_id=WQDVG2KXSEDZHJJOT2ZFSAPQXVLJZXMO5HQXXEU6XJ7U7CAQWZVQ, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.462301153 (563h51m7.333384198s) [noise=true , address=145.79.227.21:4498 , piece_id=RTK3PTWGWTIQSNCGREQB37OID2RGK37QWEKTDYFQKASGH6SZCURA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.479891718 (563h51m7.350974763s) [noise=true , address=145.79.227.12:4778 , piece_id=KHBVVV3WSILPU4CQK4HAUC37D75DUWGOJWZFY53SBDLVY3F3EBAA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.486062297 (563h51m7.357145342s) [noise=true , address=145.79.227.2:4142 , piece_id=GS7XTVLRWWXMZKGYSVDGKAGR6QX3OJUTK4TNNVPPA26XS5KSHKBQ, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.578667930 (563h51m7.449750975s) [noise=true , address=145.79.227.50:4391 , piece_id=GX6GJKCSCDKGMYRAPNP4Z347JJMKPWFLDT2QGOSSZGAYKNZNTDHA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.701241555 (563h51m7.5723246s) [noise=true , address=45.91.116.13:4177 , piece_id=YIJBXUI5PGKEJOY3USREMMV35V2VMXXFILDOUNJPXMIHVD5PZIUA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.745868671 (563h51m7.616951706s) [noise=true , address=77.90.31.13:4644 , piece_id=VJZEITJXVPJHT6FPHXEFDMWDGR4XQNAZMDGP7BOKEQEAKKN5LKJQ, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.799936903 (563h51m7.671019948s) [noise=true , address=77.90.31.16:4739 , piece_id=5I3LCPAI2IF5Z5MZWQNKDM7LKBYEYIPE3EBUAD6DOLBPW34FDNPA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:08.430929511 (563h51m8.302012556s) [noise=true , address=77.90.31.11:4588 , piece_id=XU7QUQZHRH27X64OR4NXJERPPIHCPBEGL7POHUL5Z32LTFMI5MKQ, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:08.432106221 (563h51m8.303189256s) [noise=true , address=145.79.227.35:4493 , piece_id=A2ZOH3MMXV5S7JPOWOCQZMYQEMF3SJHTYIIOB2656AVAXP24UAYA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:08.447633778 (563h51m8.318716813s) [noise=true , address=145.79.227.49:4355 , piece_id=ZBF32JIHB33TYP2KB427V7OQRRNXP2ZOTLZNFRQ5XUH7HGPG2RTA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:08.459472520 (563h51m8.330555555s) [noise=true , address=145.79.227.34:4916 , piece_id=4U67LINR5A4GR2NIUVMB2IPTT5DJQOYHM6HDYKZEICKF5JY53VFA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:08.499215165 (563h51m8.37029821s) [noise=true , address=145.79.227.43:4170 , piece_id=F7H3IF2SNNGTIV2CFKXB3GIA6TGRLPSOLA6TPBUW5FQVWULEK4GQ, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:08.499245653 (563h51m8.370328698s) [seg_pos={0 0}, upload=494362]: Segment reached optimal threshold of 14 pieces. Dec 29 03:20:08.499544722 (563h51m8.370627797s) [noise=true , address=145.79.227.35:4507 , piece_id=SNS3BVKFWUKQYMNZCWMOHH563GXNM5IIB2QMLGVCCMIXHEV7MJOA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14jurWWPpmxwiBS3XmZysEUwugsceYJqVXFj7BbYpx8UFcTR8R): protocol: expected piece hash; context canceled; EOF Dec 29 03:20:08.499555243 (563h51m8.370638288s) [noise=true , address=145.79.227.35:4507 , piece_id=SNS3BVKFWUKQYMNZCWMOHH563GXNM5IIB2QMLGVCCMIXHEV7MJOA, seg_pos={0 0}, upload=494362]: Long tail cancellation due to optimal threshold Dec 29 03:20:08.499833272 (563h51m8.370916327s) [noise=true , address=77.90.31.18:4838 , piece_id=TP42RP5AUZSP6LNKK7ON4FMDFVF6GRX3FLRYWBS56SCKFAZNPQSA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1if7vf1cNTBwAhb6owi6tuGLYqSfa3feWhsvGSevwuwykhz2RY): protocol: write tcp 188.166.225.170:39578->77.90.31.18:4838: use of closed network connection; write tcp 188.166.225.170:39578->77.90.31.18:4838: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39578->77.90.31.18:4838: use of closed network connection Dec 29 03:20:08.499843291 (563h51m8.370926366s) [noise=true , address=77.90.31.18:4838 , piece_id=TP42RP5AUZSP6LNKK7ON4FMDFVF6GRX3FLRYWBS56SCKFAZNPQSA, seg_pos={0 0}, upload=494362]: Long tail cancellation due to optimal threshold Dec 29 03:20:08.500107964 (563h51m8.371191019s) [noise=true , address=77.90.31.21:4909 , piece_id=KKTXOQ647SWOMJ7J4DC6CMHBSI65JJD2C5IMZZWI3ZSQI7SXLY2A, seg_pos={0 0}, upload=494362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121RT7tP6czMFAiZ3DjzSSsYgPt3xZXVbXVpkcYxaZYscNiMjL6): protocol: expected piece hash; context canceled; EOF Dec 29 03:20:08.500166165 (563h51m8.37124921s) [noise=true , address=77.90.31.21:4909 , piece_id=KKTXOQ647SWOMJ7J4DC6CMHBSI65JJD2C5IMZZWI3ZSQI7SXLY2A, seg_pos={0 0}, upload=494362]: Long tail cancellation due to optimal threshold Dec 29 03:20:08.500868092 (563h51m8.371951147s) [noise=true , address=77.90.31.12:4605 , piece_id=CEYXIJYUE4MGBAOGMKOWIQZBRFRC23FFUZCP5AW7N4MCT76B5DWQ, seg_pos={0 0}, upload=494362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V4Dj8k4kAnrS6GfSJvvDDPvyHTiJziufBCWWjNoPe1Yvc6M53): protocol: context canceled; context canceled Dec 29 03:20:08.500880466 (563h51m8.371963511s) [noise=true , address=77.90.31.12:4605 , piece_id=CEYXIJYUE4MGBAOGMKOWIQZBRFRC23FFUZCP5AW7N4MCT76B5DWQ, seg_pos={0 0}, upload=494362]: Long tail cancellation due to optimal threshold Dec 29 03:20:08.500945178 (563h51m8.372028223s) [seg_pos={0 0}, upload=494362]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 03:20:08.501128358 (563h51m8.372211413s) [upload=494362]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 03:20:08.501156331 (563h51m8.372239376s) [upload=494362]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 03:20:08.795508553 (563h51m8.666591598s) [upload=494362]: Done uploading segments... Dec 29 03:20:08.795532639 (563h51m8.666615684s) [upload=494362]: Upload finished. err: <nil> Duration: 56.232869192s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 28 out of 60 is failed (threshold=3) [X.XX..XX.....XX.X.....X.X..XXXX..XXX.XX...X.X.....XXX...XXXX] duration: 30.003147959s trace: https://jaeger.dev.storj.io/trace/1411f8ba3ffad9d5 FAIL (2025-12-29 03:37:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, Piece ID: SXTAMUTZVMRWPJHQRAYXM4FLHZWJTRYDDAEDOACWDXHJLLHZLFHA) context deadline exceeded; piecestore: (Node ID: 123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, Piece ID: KRYZPON3V3SSNRBLUHA76XKHT2SPFHHBX5KEQVQG3QEAGFCQLR6A) context deadline exceeded; piecestore: (Node ID: 1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, Piece ID: YOR5TWBPXU7FXCFHDBNRUHRDBUQSSPWOCAK5CWEA3GXJGQK2RSQQ) context deadline exceeded ======LOGS====== 08]: Starting upload Dec 29 03:36:49.301462682 (564h7m49.172545717s) [upload=494608]: Done starting upload Dec 29 03:36:49.301526193 (564h7m49.172609228s) [upload=494608]: Uploading segments... Dec 29 03:36:49.301688091 (564h7m49.172771126s) [upload=494608]: Got next segment. Inline: false Dec 29 03:36:49.301752564 (564h7m49.172835609s) [upload=494608]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 03:36:49.301760018 (564h7m49.172843113s) [upload=494608]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 03:36:49.547947304 (564h7m49.419030339s) [seg_pos={0 0}, upload=494608]: Begin upload segment... Dec 29 03:36:49.548014071 (564h7m49.419097116s) [seg_pos={0 0}, upload=494608]: Done begin upload segment. Dec 29 03:36:49.548105435 (564h7m49.419188481s) [noise=true , address=145.79.227.36:4517 , piece_id=RYDCIQPCI2ADU3MQGP3EG6QJKQU2OZ74SYNLRU75ZVSDX3G46WOQ, seg_pos={0 0}, upload=494608]: Uploading piece... Dec 29 03:36:49.548123489 (564h7m49.419206524s) [noise=true , address=145.79.227.39:4015 , piece_id=HCHKGBMSYMR5JCL5YNYCGWX7U3UYFRZH6N3D2BUVLPP5Y35LADFQ, seg_pos={0 0}, upload=494608]: Uploading piece... Dec 29 03:36:49.548188363 (564h7m49.419271398s) [noise=true , address=145.79.227.37:4567 , piece_id=VQU6DWRUHYWVYDKFK3WNAKABDJQTSKCZXCHB2U2OM6E3V3SCD52A, seg_pos={0 0}, upload=494608]: Uploading piece... Dec 29 03:36:49.548825607 (564h7m49.419908642s) [noise=true , address=145.79.227.2:4113 , piece_id=METUMZTP3SQYMS4ZUSXVQIE6Q5L6C7UEJTCSTQQMXEN2246LQ4VA, seg_pos={0 0}, upload=494608]: Uploading piece... Dec 29 03:36:49.549249264 (564h7m49.420332299s) [noise=true , address=45.91.116.5:4298 , piece_id=SXTAMUTZVMRWPJHQRAYXM4FLHZWJTRYDDAEDOACWDXHJLLHZLFHA, seg_pos={0 0}, upload=494608]: Uploading piece... Dec 29 03:36:49.549679062 (564h7m49.420762107s) [noise=true , address=45.91.116.9:4262 , piece_id=2YMSN24ST7AHPJWOF3JGPS4W4HWSXLGP7HWDPGSIKJ3OVFJWOHBQ, seg_pos={0 0}, upload=494608]: Uploading piece... Dec 29 03:36:49.550209101 (564h7m49.421292146s) [noise=true , address=45.91.116.10:4083 , piece_id=KRYZPON3V3SSNRBLUHA76XKHT2SPFHHBX5KEQVQG3QEAGFCQLR6A, seg_pos={0 0}, upload=494608]: Uploading piece... Dec 29 03:36:49.550729522 (564h7m49.421812557s) [noise=true , address=77.90.31.11:4554 , piece_id=YOR5TWBPXU7FXCFHDBNRUHRDBUQSSPWOCAK5CWEA3GXJGQK2RSQQ, seg_pos={0 0}, upload=494608]: Uploading piece... Dec 29 03:36:49.551239493 (564h7m49.422322538s) [noise=true , address=145.79.227.46:4601 , piece_id=FRO6WKYZO565BATDK2EI2BEFNHOJUOOSCE2ZQ6EJ2D3PYRXVQR2A, seg_pos={0 0}, upload=494608]: Uploading piece... Dec 29 03:36:49.551720660 (564h7m49.422803695s) [noise=true , address=145.79.227.45:4230 , piece_id=TNLHZXR4KNWTEKCMN5YRD3TEECDIVZMBTEIMPMNDDWNDQQVV6AOA, seg_pos={0 0}, upload=494608]: Uploading piece... Dec 29 03:36:49.552210913 (564h7m49.423293958s) [noise=true , address=45.91.116.4:4940 , piece_id=ZCNRQMFHXGQEXUMF226F6Y63NRB4GEW56AAXLN7BPPYGICYHIQWA, seg_pos={0 0}, upload=494608]: Uploading piece... Dec 29 03:36:49.553436116 (564h7m49.424519161s) [noise=true , address=145.79.227.32:4835 , piece_id=5J7NKVRPAN7CCPMC4RAAWH62Z3OF3QZIOOMP3JI6M6QHZHIK6TPA, seg_pos={0 0}, upload=494608]: Uploading piece... Dec 29 03:36:49.553744994 (564h7m49.424828039s) [noise=true , address=45.91.116.6:4974 , piece_id=ZDZZ6S6EBVNQQE7BSHC2J2XI6ZFPXMTMOOGAEVEA35BDQZTIAILA, seg_pos={0 0}, upload=494608]: Uploading piece... Dec 29 03:36:49.555323159 (564h7m49.426406194s) [noise=true , address=77.90.31.17:4773 , piece_id=3M3PHGM3YDZOMA7ALU3WAJTIZZCD2DW7ZIJKAN5F65ZVX7YCYAJA, seg_pos={0 0}, upload=494608]: Uploading piece... Dec 29 03:36:49.555468135 (564h7m49.42655117s) [noise=true , address=145.79.227.11:4732 , piece_id=UXLX7JE23UR27CUSBQCXWKLVVDNON7MPTLYQQXE7UYQ2X6SF2AJA, seg_pos={0 0}, upload=494608]: Uploading piece... Dec 29 03:36:49.557144577 (564h7m49.428227622s) [noise=true , address=145.79.227.13:4802 , piece_id=6ADJZ2RTSJLXU2KPZ5UPGNMJCFFMQL5O2AWKKUC2JBK6HSC2IRXA, seg_pos={0 0}, upload=494608]: Uploading piece... Dec 29 03:36:51.254589086 (564h7m51.125672131s) [upload=494608]: Next returned nil segment Dec 29 03:36:51.254602902 (564h7m51.125685937s) [upload=494608]: Waiting for error group managing segments... Dec 29 03:36:51.535758903 (564h7m51.406841938s) [noise=true , address=145.79.227.37:4567 , piece_id=VQU6DWRUHYWVYDKFK3WNAKABDJQTSKCZXCHB2U2OM6E3V3SCD52A, seg_pos={0 0}, upload=494608]: Done uploading piece. err: <nil> Dec 29 03:36:51.557049213 (564h7m51.428132248s) [noise=true , address=145.79.227.45:4230 , piece_id=TNLHZXR4KNWTEKCMN5YRD3TEECDIVZMBTEIMPMNDDWNDQQVV6AOA, seg_pos={0 0}, upload=494608]: Done uploading piece. err: <nil> Dec 29 03:36:51.557108195 (564h7m51.428191241s) [noise=true , address=145.79.227.32:4835 , piece_id=5J7NKVRPAN7CCPMC4RAAWH62Z3OF3QZIOOMP3JI6M6QHZHIK6TPA, seg_pos={0 0}, upload=494608]: Done uploading piece. err: <nil> Dec 29 03:36:51.568152976 (564h7m51.439236021s) [noise=true , address=145.79.227.13:4802 , piece_id=6ADJZ2RTSJLXU2KPZ5UPGNMJCFFMQL5O2AWKKUC2JBK6HSC2IRXA, seg_pos={0 0}, upload=494608]: Done uploading piece. err: <nil> Dec 29 03:36:51.573632841 (564h7m51.444715886s) [noise=true , address=145.79.227.39:4015 , piece_id=HCHKGBMSYMR5JCL5YNYCGWX7U3UYFRZH6N3D2BUVLPP5Y35LADFQ, seg_pos={0 0}, upload=494608]: Done uploading piece. err: <nil> Dec 29 03:36:51.575501148 (564h7m51.446584194s) [noise=true , address=145.79.227.46:4601 , piece_id=FRO6WKYZO565BATDK2EI2BEFNHOJUOOSCE2ZQ6EJ2D3PYRXVQR2A, seg_pos={0 0}, upload=494608]: Done uploading piece. err: <nil> Dec 29 03:36:51.576209948 (564h7m51.447292983s) [noise=true , address=145.79.227.36:4517 , piece_id=RYDCIQPCI2ADU3MQGP3EG6QJKQU2OZ74SYNLRU75ZVSDX3G46WOQ, seg_pos={0 0}, upload=494608]: Done uploading piece. err: <nil> Dec 29 03:36:51.597716239 (564h7m51.468799274s) [noise=true , address=145.79.227.11:4732 , piece_id=UXLX7JE23UR27CUSBQCXWKLVVDNON7MPTLYQQXE7UYQ2X6SF2AJA, seg_pos={0 0}, upload=494608]: Done uploading piece. err: <nil> Dec 29 03:36:51.863055223 (564h7m51.734138258s) [noise=true , address=45.91.116.10:4083 , piece_id=KRYZPON3V3SSNRBLUHA76XKHT2SPFHHBX5KEQVQG3QEAGFCQLR6A, seg_pos={0 0}, upload=494608]: Done uploading piece. err: <nil> Dec 29 03:36:51.924048513 (564h7m51.795131548s) [noise=true , address=145.79.227.2:4113 , piece_id=METUMZTP3SQYMS4ZUSXVQIE6Q5L6C7UEJTCSTQQMXEN2246LQ4VA, seg_pos={0 0}, upload=494608]: Done uploading piece. err: <nil> Dec 29 03:36:51.952469681 (564h7m51.823552716s) [noise=true , address=45.91.116.5:4298 , piece_id=SXTAMUTZVMRWPJHQRAYXM4FLHZWJTRYDDAEDOACWDXHJLLHZLFHA, seg_pos={0 0}, upload=494608]: Done uploading piece. err: <nil> Dec 29 03:36:52.116013461 (564h7m51.987096506s) [noise=true , address=77.90.31.11:4554 , piece_id=YOR5TWBPXU7FXCFHDBNRUHRDBUQSSPWOCAK5CWEA3GXJGQK2RSQQ, seg_pos={0 0}, upload=494608]: Done uploading piece. err: <nil> Dec 29 03:36:52.267267800 (564h7m52.138350835s) [noise=true , address=45.91.116.9:4262 , piece_id=2YMSN24ST7AHPJWOF3JGPS4W4HWSXLGP7HWDPGSIKJ3OVFJWOHBQ, seg_pos={0 0}, upload=494608]: Done uploading piece. err: <nil> Dec 29 03:36:52.267289492 (564h7m52.138372527s) [seg_pos={0 0}, upload=494608]: Segment reached optimal threshold of 13 pieces. Dec 29 03:36:52.267485836 (564h7m52.138568871s) [noise=true , address=77.90.31.17:4773 , piece_id=3M3PHGM3YDZOMA7ALU3WAJTIZZCD2DW7ZIJKAN5F65ZVX7YCYAJA, seg_pos={0 0}, upload=494608]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ): protocol: write tcp 188.166.225.170:50660->77.90.31.17:4773: use of closed network connection; write tcp 188.166.225.170:50660->77.90.31.17:4773: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50660->77.90.31.17:4773: use of closed network connection Dec 29 03:36:52.267491437 (564h7m52.138574492s) [noise=true , address=77.90.31.17:4773 , piece_id=3M3PHGM3YDZOMA7ALU3WAJTIZZCD2DW7ZIJKAN5F65ZVX7YCYAJA, seg_pos={0 0}, upload=494608]: Long tail cancellation due to optimal threshold Dec 29 03:36:52.267707478 (564h7m52.138790513s) [noise=true , address=45.91.116.4:4940 , piece_id=ZCNRQMFHXGQEXUMF226F6Y63NRB4GEW56AAXLN7BPPYGICYHIQWA, seg_pos={0 0}, upload=494608]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: read tcp 188.166.225.170:39836->45.91.116.4:4940: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 29 03:36:52.267713650 (564h7m52.138796685s) [noise=true , address=45.91.116.4:4940 , piece_id=ZCNRQMFHXGQEXUMF226F6Y63NRB4GEW56AAXLN7BPPYGICYHIQWA, seg_pos={0 0}, upload=494608]: Long tail cancellation due to optimal threshold Dec 29 03:36:52.267944780 (564h7m52.139027815s) [noise=true , address=45.91.116.6:4974 , piece_id=ZDZZ6S6EBVNQQE7BSHC2J2XI6ZFPXMTMOOGAEVEA35BDQZTIAILA, seg_pos={0 0}, upload=494608]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: expected piece hash; context canceled; EOF Dec 29 03:36:52.267964998 (564h7m52.139048033s) [noise=true , address=45.91.116.6:4974 , piece_id=ZDZZ6S6EBVNQQE7BSHC2J2XI6ZFPXMTMOOGAEVEA35BDQZTIAILA, seg_pos={0 0}, upload=494608]: Long tail cancellation due to optimal threshold Dec 29 03:36:52.267999644 (564h7m52.139082679s) [seg_pos={0 0}, upload=494608]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 03:36:52.268160901 (564h7m52.139243946s) [upload=494608]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 03:36:52.268169918 (564h7m52.139252953s) [upload=494608]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 03:36:52.555582033 (564h7m52.426665068s) [upload=494608]: Done uploading segments... Dec 29 03:36:52.555601900 (564h7m52.426684935s) [upload=494608]: Upload finished. err: <nil> Duration: 30.003172755s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [.....................................XX................X....] duration: 3.739559843s trace: https://jaeger.dev.storj.io/trace/611c7ac3f284e077 FAIL (2025-12-29 03:36:48 UTC) orbiter warning: context deadline exceeded Duration: 3.739585712s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 41.548921606s trace: https://jaeger.dev.storj.io/trace/1584175ba8dba7fe FAIL (2025-12-29 03:36:26 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ece... Dec 29 03:20:05.592697598 (563h51m5.463780643s) [noise=true , address=77.90.31.16:4739 , piece_id=5I3LCPAI2IF5Z5MZWQNKDM7LKBYEYIPE3EBUAD6DOLBPW34FDNPA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.593249339 (563h51m5.464332384s) [noise=true , address=77.90.31.12:4605 , piece_id=CEYXIJYUE4MGBAOGMKOWIQZBRFRC23FFUZCP5AW7N4MCT76B5DWQ, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.593687443 (563h51m5.464770488s) [noise=true , address=145.79.227.49:4355 , piece_id=ZBF32JIHB33TYP2KB427V7OQRRNXP2ZOTLZNFRQ5XUH7HGPG2RTA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.594173598 (563h51m5.465256633s) [noise=true , address=145.79.227.2:4142 , piece_id=GS7XTVLRWWXMZKGYSVDGKAGR6QX3OJUTK4TNNVPPA26XS5KSHKBQ, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.594639957 (563h51m5.465723002s) [noise=true , address=145.79.227.9:4382 , piece_id=WQDVG2KXSEDZHJJOT2ZFSAPQXVLJZXMO5HQXXEU6XJ7U7CAQWZVQ, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.595331994 (563h51m5.466415039s) [noise=true , address=145.79.227.43:4170 , piece_id=F7H3IF2SNNGTIV2CFKXB3GIA6TGRLPSOLA6TPBUW5FQVWULEK4GQ, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.595834151 (563h51m5.466917206s) [noise=true , address=145.79.227.12:4778 , piece_id=KHBVVV3WSILPU4CQK4HAUC37D75DUWGOJWZFY53SBDLVY3F3EBAA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.596329844 (563h51m5.467412899s) [noise=true , address=145.79.227.50:4391 , piece_id=GX6GJKCSCDKGMYRAPNP4Z347JJMKPWFLDT2QGOSSZGAYKNZNTDHA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.599128643 (563h51m5.470211688s) [noise=true , address=45.91.116.13:4177 , piece_id=YIJBXUI5PGKEJOY3USREMMV35V2VMXXFILDOUNJPXMIHVD5PZIUA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.599196953 (563h51m5.470279988s) [noise=true , address=145.79.227.35:4493 , piece_id=A2ZOH3MMXV5S7JPOWOCQZMYQEMF3SJHTYIIOB2656AVAXP24UAYA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.601094367 (563h51m5.472177412s) [noise=true , address=77.90.31.18:4838 , piece_id=TP42RP5AUZSP6LNKK7ON4FMDFVF6GRX3FLRYWBS56SCKFAZNPQSA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.602868896 (563h51m5.473951941s) [noise=true , address=77.90.31.21:4909 , piece_id=KKTXOQ647SWOMJ7J4DC6CMHBSI65JJD2C5IMZZWI3ZSQI7SXLY2A, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.604661328 (563h51m5.475744374s) [noise=true , address=77.90.31.13:4644 , piece_id=VJZEITJXVPJHT6FPHXEFDMWDGR4XQNAZMDGP7BOKEQEAKKN5LKJQ, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.606462138 (563h51m5.477545183s) [noise=true , address=145.79.227.35:4507 , piece_id=SNS3BVKFWUKQYMNZCWMOHH563GXNM5IIB2QMLGVCCMIXHEV7MJOA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.609580124 (563h51m5.480663169s) [noise=true , address=64.49.13.140:4515 , piece_id=BB4RND6DQPMXBJHNTIFIQWVZOFZFHIFB6KBYEUOKOLJFDCWRIMJA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.611644807 (563h51m5.482727852s) [noise=true , address=77.90.31.11:4588 , piece_id=XU7QUQZHRH27X64OR4NXJERPPIHCPBEGL7POHUL5Z32LTFMI5MKQ, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:06.901416710 (563h51m6.772499755s) [upload=494362]: Next returned nil segment Dec 29 03:20:06.901434123 (563h51m6.772517168s) [upload=494362]: Waiting for error group managing segments... Dec 29 03:20:07.237383793 (563h51m7.108466848s) [noise=true , address=64.49.13.140:4515 , piece_id=BB4RND6DQPMXBJHNTIFIQWVZOFZFHIFB6KBYEUOKOLJFDCWRIMJA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.439111607 (563h51m7.310194652s) [noise=true , address=145.79.227.9:4382 , piece_id=WQDVG2KXSEDZHJJOT2ZFSAPQXVLJZXMO5HQXXEU6XJ7U7CAQWZVQ, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.462301153 (563h51m7.333384198s) [noise=true , address=145.79.227.21:4498 , piece_id=RTK3PTWGWTIQSNCGREQB37OID2RGK37QWEKTDYFQKASGH6SZCURA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.479891718 (563h51m7.350974763s) [noise=true , address=145.79.227.12:4778 , piece_id=KHBVVV3WSILPU4CQK4HAUC37D75DUWGOJWZFY53SBDLVY3F3EBAA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.486062297 (563h51m7.357145342s) [noise=true , address=145.79.227.2:4142 , piece_id=GS7XTVLRWWXMZKGYSVDGKAGR6QX3OJUTK4TNNVPPA26XS5KSHKBQ, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.578667930 (563h51m7.449750975s) [noise=true , address=145.79.227.50:4391 , piece_id=GX6GJKCSCDKGMYRAPNP4Z347JJMKPWFLDT2QGOSSZGAYKNZNTDHA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.701241555 (563h51m7.5723246s) [noise=true , address=45.91.116.13:4177 , piece_id=YIJBXUI5PGKEJOY3USREMMV35V2VMXXFILDOUNJPXMIHVD5PZIUA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.745868671 (563h51m7.616951706s) [noise=true , address=77.90.31.13:4644 , piece_id=VJZEITJXVPJHT6FPHXEFDMWDGR4XQNAZMDGP7BOKEQEAKKN5LKJQ, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.799936903 (563h51m7.671019948s) [noise=true , address=77.90.31.16:4739 , piece_id=5I3LCPAI2IF5Z5MZWQNKDM7LKBYEYIPE3EBUAD6DOLBPW34FDNPA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:08.430929511 (563h51m8.302012556s) [noise=true , address=77.90.31.11:4588 , piece_id=XU7QUQZHRH27X64OR4NXJERPPIHCPBEGL7POHUL5Z32LTFMI5MKQ, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:08.432106221 (563h51m8.303189256s) [noise=true , address=145.79.227.35:4493 , piece_id=A2ZOH3MMXV5S7JPOWOCQZMYQEMF3SJHTYIIOB2656AVAXP24UAYA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:08.447633778 (563h51m8.318716813s) [noise=true , address=145.79.227.49:4355 , piece_id=ZBF32JIHB33TYP2KB427V7OQRRNXP2ZOTLZNFRQ5XUH7HGPG2RTA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:08.459472520 (563h51m8.330555555s) [noise=true , address=145.79.227.34:4916 , piece_id=4U67LINR5A4GR2NIUVMB2IPTT5DJQOYHM6HDYKZEICKF5JY53VFA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:08.499215165 (563h51m8.37029821s) [noise=true , address=145.79.227.43:4170 , piece_id=F7H3IF2SNNGTIV2CFKXB3GIA6TGRLPSOLA6TPBUW5FQVWULEK4GQ, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:08.499245653 (563h51m8.370328698s) [seg_pos={0 0}, upload=494362]: Segment reached optimal threshold of 14 pieces. Dec 29 03:20:08.499544722 (563h51m8.370627797s) [noise=true , address=145.79.227.35:4507 , piece_id=SNS3BVKFWUKQYMNZCWMOHH563GXNM5IIB2QMLGVCCMIXHEV7MJOA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14jurWWPpmxwiBS3XmZysEUwugsceYJqVXFj7BbYpx8UFcTR8R): protocol: expected piece hash; context canceled; EOF Dec 29 03:20:08.499555243 (563h51m8.370638288s) [noise=true , address=145.79.227.35:4507 , piece_id=SNS3BVKFWUKQYMNZCWMOHH563GXNM5IIB2QMLGVCCMIXHEV7MJOA, seg_pos={0 0}, upload=494362]: Long tail cancellation due to optimal threshold Dec 29 03:20:08.499833272 (563h51m8.370916327s) [noise=true , address=77.90.31.18:4838 , piece_id=TP42RP5AUZSP6LNKK7ON4FMDFVF6GRX3FLRYWBS56SCKFAZNPQSA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1if7vf1cNTBwAhb6owi6tuGLYqSfa3feWhsvGSevwuwykhz2RY): protocol: write tcp 188.166.225.170:39578->77.90.31.18:4838: use of closed network connection; write tcp 188.166.225.170:39578->77.90.31.18:4838: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39578->77.90.31.18:4838: use of closed network connection Dec 29 03:20:08.499843291 (563h51m8.370926366s) [noise=true , address=77.90.31.18:4838 , piece_id=TP42RP5AUZSP6LNKK7ON4FMDFVF6GRX3FLRYWBS56SCKFAZNPQSA, seg_pos={0 0}, upload=494362]: Long tail cancellation due to optimal threshold Dec 29 03:20:08.500107964 (563h51m8.371191019s) [noise=true , address=77.90.31.21:4909 , piece_id=KKTXOQ647SWOMJ7J4DC6CMHBSI65JJD2C5IMZZWI3ZSQI7SXLY2A, seg_pos={0 0}, upload=494362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121RT7tP6czMFAiZ3DjzSSsYgPt3xZXVbXVpkcYxaZYscNiMjL6): protocol: expected piece hash; context canceled; EOF Dec 29 03:20:08.500166165 (563h51m8.37124921s) [noise=true , address=77.90.31.21:4909 , piece_id=KKTXOQ647SWOMJ7J4DC6CMHBSI65JJD2C5IMZZWI3ZSQI7SXLY2A, seg_pos={0 0}, upload=494362]: Long tail cancellation due to optimal threshold Dec 29 03:20:08.500868092 (563h51m8.371951147s) [noise=true , address=77.90.31.12:4605 , piece_id=CEYXIJYUE4MGBAOGMKOWIQZBRFRC23FFUZCP5AW7N4MCT76B5DWQ, seg_pos={0 0}, upload=494362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V4Dj8k4kAnrS6GfSJvvDDPvyHTiJziufBCWWjNoPe1Yvc6M53): protocol: context canceled; context canceled Dec 29 03:20:08.500880466 (563h51m8.371963511s) [noise=true , address=77.90.31.12:4605 , piece_id=CEYXIJYUE4MGBAOGMKOWIQZBRFRC23FFUZCP5AW7N4MCT76B5DWQ, seg_pos={0 0}, upload=494362]: Long tail cancellation due to optimal threshold Dec 29 03:20:08.500945178 (563h51m8.372028223s) [seg_pos={0 0}, upload=494362]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 03:20:08.501128358 (563h51m8.372211413s) [upload=494362]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 03:20:08.501156331 (563h51m8.372239376s) [upload=494362]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 03:20:08.795508553 (563h51m8.666591598s) [upload=494362]: Done uploading segments... Dec 29 03:20:08.795532639 (563h51m8.666615684s) [upload=494362]: Upload finished. err: <nil> Duration: 41.548954588s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)