Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 55 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXX..X.XXXX.XXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXX] duration: 5.000964785s trace: https://jaeger.dev.storj.io/trace/4e44a8b9c3ec5f38 FAIL (2025-12-10 13:34:42 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000990303s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 7.579448653s trace: https://jaeger.dev.storj.io/trace/a0b5a94505b001 FAIL (2025-12-10 13:34:31 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:36068->45.38.24.2:443: read: connection reset by peer Duration: 7.579503317s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 9.63098094s trace: https://jaeger.dev.storj.io/trace/19144dc231ae8183 FAIL (2025-12-10 13:34:26 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ts... Dec 10 13:15:32.674883550 (117h46m32.545966595s) [upload=102799]: Got next segment. Inline: false Dec 10 13:15:32.674940880 (117h46m32.546023915s) [upload=102799]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 10 13:15:32.674946670 (117h46m32.546029706s) [upload=102799]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 10 13:15:33.036416584 (117h46m32.907499629s) [seg_pos={0 0}, upload=102799]: Begin upload segment... Dec 10 13:15:33.036472831 (117h46m32.907555876s) [seg_pos={0 0}, upload=102799]: Done begin upload segment. Dec 10 13:15:33.036565578 (117h46m32.907648613s) [noise=true , address=145.79.227.29:4943 , piece_id=I5NHKVGJKAG37VAV53QOQ2ZYSDF6SGEXTTG7DEAAG5CC2DO7SVVQ, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.036606606 (117h46m32.907689651s) [noise=true , address=145.79.227.6:4259 , piece_id=RQCMHQGVQGAWFBJ6AAHMVON7SLBV7XPYR4D6XRI6DJEJVJ7AFGRQ, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.037385819 (117h46m32.908468865s) [noise=true , address=145.79.227.26:4654 , piece_id=TRV36WT3M3K5LIJB3LCMLG6OFPQWWOMYHFOHGKLRECB7NE5VBEGA, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.037833743 (117h46m32.908916778s) [noise=true , address=145.79.227.22:4509 , piece_id=62EZ3JKY7O5VZEGSATQYYBLKR46ASNJUX3H7YIYAIXAV4WYF26JQ, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.038339726 (117h46m32.909422772s) [noise=true , address=145.79.227.10:4401 , piece_id=S4DH6HYJ5IUWJ5BTURVW7OTDWFS3O3UNCZMSTUK4QZ4OCNKTH6MA, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.038758464 (117h46m32.909841509s) [noise=true , address=145.79.227.40:4049 , piece_id=FVAQGZN6T3VF2XVZB4PCEYGS3PU3XGLBSN4FLFQTYIEC5HDTC3OQ, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.039163404 (117h46m32.910246449s) [noise=true , address=77.90.31.17:4771 , piece_id=X57ZVYWUTGUUKAOIRL2FL5CHXAN5SQYLX6EJBQIKL4FZGHWGPRQA, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.039588294 (117h46m32.910671339s) [noise=true , address=145.79.227.39:4014 , piece_id=HO5AOQJRC774MOI5S6J7MOHIYV4L72XC22HLQCHBLYP3QYIA2DTQ, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.040016559 (117h46m32.911099594s) [noise=true , address=145.79.227.43:4159 , piece_id=5KI6NRQSBBPKKUN65NU5LTY32YJ65K2OKZDJQEVQX465WNQKNHHQ, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.040450956 (117h46m32.911534001s) [noise=true , address=77.90.31.10:4518 , piece_id=XCU5D2A36LBTDDYTILXWUMMANHVDCW2KMEAB3HUCDU5UONZLULBQ, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.040890654 (117h46m32.911973689s) [noise=true , address=145.79.227.22:4510 , piece_id=ALCQVB6CUD7JAJ7I2NHADFKGHOSFVBKCGR2KHZWHPMYKK3VV4T5A, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.041348756 (117h46m32.912431801s) [noise=true , address=45.91.116.11:4119 , piece_id=S5KLL473HCDKQBM5NVTVNLLSYGMXFFTFDFQX7D5SITM2MQO43K6Q, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.041761201 (117h46m32.912844236s) [noise=true , address=145.79.227.41:4087 , piece_id=OQCN6TONOCOR2EH7THCYREB7LRVYOMK55VPTAWK4NMADKYYYPYLA, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.042078305 (117h46m32.91316134s) [noise=true , address=145.79.227.7:4295 , piece_id=IPMI23BRBMF4MTC6EJJEZWGW7AMFPLGUGMXEFCF2QZASWEUFIH5Q, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.042443110 (117h46m32.913526155s) [noise=true , address=145.79.227.32:4835 , piece_id=4X2XQSZAB6KZPDY5G4VFDFZCL744RAL2IYUYMU7UWFMEII5D225A, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.042561475 (117h46m32.91364452s) [noise=true , address=145.79.227.31:4797 , piece_id=WOGJMBDWY42R5T5T5M2NR4AF3NFZ6CAYOI2O5RRL3JYPCAB3FFZA, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:35.033962808 (117h46m34.905045853s) [upload=102799]: Next returned nil segment Dec 10 13:15:35.033993557 (117h46m34.905076612s) [upload=102799]: Waiting for error group managing segments... Dec 10 13:15:35.355027663 (117h46m35.226110708s) [noise=true , address=77.90.31.10:4518 , piece_id=XCU5D2A36LBTDDYTILXWUMMANHVDCW2KMEAB3HUCDU5UONZLULBQ, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:35.395849857 (117h46m35.266932902s) [noise=true , address=77.90.31.17:4771 , piece_id=X57ZVYWUTGUUKAOIRL2FL5CHXAN5SQYLX6EJBQIKL4FZGHWGPRQA, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:35.939806675 (117h46m35.810889721s) [noise=true , address=145.79.227.31:4797 , piece_id=WOGJMBDWY42R5T5T5M2NR4AF3NFZ6CAYOI2O5RRL3JYPCAB3FFZA, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:35.944824321 (117h46m35.815907356s) [noise=true , address=145.79.227.7:4295 , piece_id=IPMI23BRBMF4MTC6EJJEZWGW7AMFPLGUGMXEFCF2QZASWEUFIH5Q, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:35.966925235 (117h46m35.83800827s) [noise=true , address=145.79.227.41:4087 , piece_id=OQCN6TONOCOR2EH7THCYREB7LRVYOMK55VPTAWK4NMADKYYYPYLA, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:36.183487725 (117h46m36.05457076s) [noise=true , address=145.79.227.29:4943 , piece_id=I5NHKVGJKAG37VAV53QOQ2ZYSDF6SGEXTTG7DEAAG5CC2DO7SVVQ, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:36.184533436 (117h46m36.055616471s) [noise=true , address=145.79.227.22:4509 , piece_id=62EZ3JKY7O5VZEGSATQYYBLKR46ASNJUX3H7YIYAIXAV4WYF26JQ, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:36.188784180 (117h46m36.059867215s) [noise=true , address=145.79.227.6:4259 , piece_id=RQCMHQGVQGAWFBJ6AAHMVON7SLBV7XPYR4D6XRI6DJEJVJ7AFGRQ, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:36.190871565 (117h46m36.0619546s) [noise=true , address=145.79.227.22:4510 , piece_id=ALCQVB6CUD7JAJ7I2NHADFKGHOSFVBKCGR2KHZWHPMYKK3VV4T5A, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:36.234924680 (117h46m36.106007725s) [noise=true , address=145.79.227.39:4014 , piece_id=HO5AOQJRC774MOI5S6J7MOHIYV4L72XC22HLQCHBLYP3QYIA2DTQ, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:37.136102891 (117h46m37.007185926s) [noise=true , address=145.79.227.32:4835 , piece_id=4X2XQSZAB6KZPDY5G4VFDFZCL744RAL2IYUYMU7UWFMEII5D225A, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:37.155928563 (117h46m37.027011618s) [noise=true , address=145.79.227.43:4159 , piece_id=5KI6NRQSBBPKKUN65NU5LTY32YJ65K2OKZDJQEVQX465WNQKNHHQ, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:37.199763735 (117h46m37.0708468s) [noise=true , address=145.79.227.10:4401 , piece_id=S4DH6HYJ5IUWJ5BTURVW7OTDWFS3O3UNCZMSTUK4QZ4OCNKTH6MA, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:37.199789233 (117h46m37.070872279s) [seg_pos={0 0}, upload=102799]: Segment reached optimal threshold of 13 pieces. Dec 10 13:15:37.200041654 (117h46m37.071124689s) [noise=true , address=145.79.227.26:4654 , piece_id=TRV36WT3M3K5LIJB3LCMLG6OFPQWWOMYHFOHGKLRECB7NE5VBEGA, seg_pos={0 0}, upload=102799]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KhBWMerZe9tUmzdVEYkBb6NmJ25dGP26Ea7CLjNJT5gdAeiah): protocol: write tcp 188.166.225.170:53216->145.79.227.26:4654: use of closed network connection; write tcp 188.166.225.170:53216->145.79.227.26:4654: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53216->145.79.227.26:4654: use of closed network connection Dec 10 13:15:37.200055591 (117h46m37.071138656s) [noise=true , address=145.79.227.26:4654 , piece_id=TRV36WT3M3K5LIJB3LCMLG6OFPQWWOMYHFOHGKLRECB7NE5VBEGA, seg_pos={0 0}, upload=102799]: Long tail cancellation due to optimal threshold Dec 10 13:15:37.200156243 (117h46m37.071239288s) [noise=true , address=145.79.227.40:4049 , piece_id=FVAQGZN6T3VF2XVZB4PCEYGS3PU3XGLBSN4FLFQTYIEC5HDTC3OQ, seg_pos={0 0}, upload=102799]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J): protocol: write tcp 188.166.225.170:55280->145.79.227.40:4049: use of closed network connection; write tcp 188.166.225.170:55280->145.79.227.40:4049: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55280->145.79.227.40:4049: use of closed network connection Dec 10 13:15:37.200184998 (117h46m37.071268053s) [noise=true , address=145.79.227.40:4049 , piece_id=FVAQGZN6T3VF2XVZB4PCEYGS3PU3XGLBSN4FLFQTYIEC5HDTC3OQ, seg_pos={0 0}, upload=102799]: Long tail cancellation due to optimal threshold Dec 10 13:15:37.200373426 (117h46m37.071456461s) [noise=true , address=45.91.116.11:4119 , piece_id=S5KLL473HCDKQBM5NVTVNLLSYGMXFFTFDFQX7D5SITM2MQO43K6Q, seg_pos={0 0}, upload=102799]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: expected piece hash; context canceled; EOF Dec 10 13:15:37.200381421 (117h46m37.071464466s) [noise=true , address=45.91.116.11:4119 , piece_id=S5KLL473HCDKQBM5NVTVNLLSYGMXFFTFDFQX7D5SITM2MQO43K6Q, seg_pos={0 0}, upload=102799]: Long tail cancellation due to optimal threshold Dec 10 13:15:37.200402562 (117h46m37.071485597s) [seg_pos={0 0}, upload=102799]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 10 13:15:37.200491951 (117h46m37.071574996s) [upload=102799]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 10 13:15:37.200500108 (117h46m37.071583153s) [upload=102799]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 10 13:15:37.624562388 (117h46m37.495645534s) [upload=102799]: Done uploading segments... Dec 10 13:15:37.624584852 (117h46m37.495667887s) [upload=102799]: Upload finished. err: <nil> Duration: 9.631015317s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [X...........................................X..X.X....X.....] duration: 7.502753368s trace: https://jaeger.dev.storj.io/trace/70ffd77b4c08e04c FAIL (2025-12-10 13:33:51 UTC) orbiter warning: context deadline exceeded Duration: 7.503060683s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 3.729527634s trace: https://jaeger.dev.storj.io/trace/a0b5a94505b001 FAIL (2025-12-10 13:33:45 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:36068->45.38.24.2:443: read: connection reset by peer Duration: 3.72955143s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 14.065497828s trace: https://jaeger.dev.storj.io/trace/19144dc231ae8183 FAIL (2025-12-10 13:33:40 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ts... Dec 10 13:15:32.674883550 (117h46m32.545966595s) [upload=102799]: Got next segment. Inline: false Dec 10 13:15:32.674940880 (117h46m32.546023915s) [upload=102799]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 10 13:15:32.674946670 (117h46m32.546029706s) [upload=102799]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 10 13:15:33.036416584 (117h46m32.907499629s) [seg_pos={0 0}, upload=102799]: Begin upload segment... Dec 10 13:15:33.036472831 (117h46m32.907555876s) [seg_pos={0 0}, upload=102799]: Done begin upload segment. Dec 10 13:15:33.036565578 (117h46m32.907648613s) [noise=true , address=145.79.227.29:4943 , piece_id=I5NHKVGJKAG37VAV53QOQ2ZYSDF6SGEXTTG7DEAAG5CC2DO7SVVQ, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.036606606 (117h46m32.907689651s) [noise=true , address=145.79.227.6:4259 , piece_id=RQCMHQGVQGAWFBJ6AAHMVON7SLBV7XPYR4D6XRI6DJEJVJ7AFGRQ, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.037385819 (117h46m32.908468865s) [noise=true , address=145.79.227.26:4654 , piece_id=TRV36WT3M3K5LIJB3LCMLG6OFPQWWOMYHFOHGKLRECB7NE5VBEGA, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.037833743 (117h46m32.908916778s) [noise=true , address=145.79.227.22:4509 , piece_id=62EZ3JKY7O5VZEGSATQYYBLKR46ASNJUX3H7YIYAIXAV4WYF26JQ, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.038339726 (117h46m32.909422772s) [noise=true , address=145.79.227.10:4401 , piece_id=S4DH6HYJ5IUWJ5BTURVW7OTDWFS3O3UNCZMSTUK4QZ4OCNKTH6MA, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.038758464 (117h46m32.909841509s) [noise=true , address=145.79.227.40:4049 , piece_id=FVAQGZN6T3VF2XVZB4PCEYGS3PU3XGLBSN4FLFQTYIEC5HDTC3OQ, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.039163404 (117h46m32.910246449s) [noise=true , address=77.90.31.17:4771 , piece_id=X57ZVYWUTGUUKAOIRL2FL5CHXAN5SQYLX6EJBQIKL4FZGHWGPRQA, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.039588294 (117h46m32.910671339s) [noise=true , address=145.79.227.39:4014 , piece_id=HO5AOQJRC774MOI5S6J7MOHIYV4L72XC22HLQCHBLYP3QYIA2DTQ, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.040016559 (117h46m32.911099594s) [noise=true , address=145.79.227.43:4159 , piece_id=5KI6NRQSBBPKKUN65NU5LTY32YJ65K2OKZDJQEVQX465WNQKNHHQ, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.040450956 (117h46m32.911534001s) [noise=true , address=77.90.31.10:4518 , piece_id=XCU5D2A36LBTDDYTILXWUMMANHVDCW2KMEAB3HUCDU5UONZLULBQ, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.040890654 (117h46m32.911973689s) [noise=true , address=145.79.227.22:4510 , piece_id=ALCQVB6CUD7JAJ7I2NHADFKGHOSFVBKCGR2KHZWHPMYKK3VV4T5A, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.041348756 (117h46m32.912431801s) [noise=true , address=45.91.116.11:4119 , piece_id=S5KLL473HCDKQBM5NVTVNLLSYGMXFFTFDFQX7D5SITM2MQO43K6Q, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.041761201 (117h46m32.912844236s) [noise=true , address=145.79.227.41:4087 , piece_id=OQCN6TONOCOR2EH7THCYREB7LRVYOMK55VPTAWK4NMADKYYYPYLA, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.042078305 (117h46m32.91316134s) [noise=true , address=145.79.227.7:4295 , piece_id=IPMI23BRBMF4MTC6EJJEZWGW7AMFPLGUGMXEFCF2QZASWEUFIH5Q, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.042443110 (117h46m32.913526155s) [noise=true , address=145.79.227.32:4835 , piece_id=4X2XQSZAB6KZPDY5G4VFDFZCL744RAL2IYUYMU7UWFMEII5D225A, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.042561475 (117h46m32.91364452s) [noise=true , address=145.79.227.31:4797 , piece_id=WOGJMBDWY42R5T5T5M2NR4AF3NFZ6CAYOI2O5RRL3JYPCAB3FFZA, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:35.033962808 (117h46m34.905045853s) [upload=102799]: Next returned nil segment Dec 10 13:15:35.033993557 (117h46m34.905076612s) [upload=102799]: Waiting for error group managing segments... Dec 10 13:15:35.355027663 (117h46m35.226110708s) [noise=true , address=77.90.31.10:4518 , piece_id=XCU5D2A36LBTDDYTILXWUMMANHVDCW2KMEAB3HUCDU5UONZLULBQ, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:35.395849857 (117h46m35.266932902s) [noise=true , address=77.90.31.17:4771 , piece_id=X57ZVYWUTGUUKAOIRL2FL5CHXAN5SQYLX6EJBQIKL4FZGHWGPRQA, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:35.939806675 (117h46m35.810889721s) [noise=true , address=145.79.227.31:4797 , piece_id=WOGJMBDWY42R5T5T5M2NR4AF3NFZ6CAYOI2O5RRL3JYPCAB3FFZA, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:35.944824321 (117h46m35.815907356s) [noise=true , address=145.79.227.7:4295 , piece_id=IPMI23BRBMF4MTC6EJJEZWGW7AMFPLGUGMXEFCF2QZASWEUFIH5Q, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:35.966925235 (117h46m35.83800827s) [noise=true , address=145.79.227.41:4087 , piece_id=OQCN6TONOCOR2EH7THCYREB7LRVYOMK55VPTAWK4NMADKYYYPYLA, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:36.183487725 (117h46m36.05457076s) [noise=true , address=145.79.227.29:4943 , piece_id=I5NHKVGJKAG37VAV53QOQ2ZYSDF6SGEXTTG7DEAAG5CC2DO7SVVQ, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:36.184533436 (117h46m36.055616471s) [noise=true , address=145.79.227.22:4509 , piece_id=62EZ3JKY7O5VZEGSATQYYBLKR46ASNJUX3H7YIYAIXAV4WYF26JQ, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:36.188784180 (117h46m36.059867215s) [noise=true , address=145.79.227.6:4259 , piece_id=RQCMHQGVQGAWFBJ6AAHMVON7SLBV7XPYR4D6XRI6DJEJVJ7AFGRQ, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:36.190871565 (117h46m36.0619546s) [noise=true , address=145.79.227.22:4510 , piece_id=ALCQVB6CUD7JAJ7I2NHADFKGHOSFVBKCGR2KHZWHPMYKK3VV4T5A, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:36.234924680 (117h46m36.106007725s) [noise=true , address=145.79.227.39:4014 , piece_id=HO5AOQJRC774MOI5S6J7MOHIYV4L72XC22HLQCHBLYP3QYIA2DTQ, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:37.136102891 (117h46m37.007185926s) [noise=true , address=145.79.227.32:4835 , piece_id=4X2XQSZAB6KZPDY5G4VFDFZCL744RAL2IYUYMU7UWFMEII5D225A, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:37.155928563 (117h46m37.027011618s) [noise=true , address=145.79.227.43:4159 , piece_id=5KI6NRQSBBPKKUN65NU5LTY32YJ65K2OKZDJQEVQX465WNQKNHHQ, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:37.199763735 (117h46m37.0708468s) [noise=true , address=145.79.227.10:4401 , piece_id=S4DH6HYJ5IUWJ5BTURVW7OTDWFS3O3UNCZMSTUK4QZ4OCNKTH6MA, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:37.199789233 (117h46m37.070872279s) [seg_pos={0 0}, upload=102799]: Segment reached optimal threshold of 13 pieces. Dec 10 13:15:37.200041654 (117h46m37.071124689s) [noise=true , address=145.79.227.26:4654 , piece_id=TRV36WT3M3K5LIJB3LCMLG6OFPQWWOMYHFOHGKLRECB7NE5VBEGA, seg_pos={0 0}, upload=102799]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KhBWMerZe9tUmzdVEYkBb6NmJ25dGP26Ea7CLjNJT5gdAeiah): protocol: write tcp 188.166.225.170:53216->145.79.227.26:4654: use of closed network connection; write tcp 188.166.225.170:53216->145.79.227.26:4654: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53216->145.79.227.26:4654: use of closed network connection Dec 10 13:15:37.200055591 (117h46m37.071138656s) [noise=true , address=145.79.227.26:4654 , piece_id=TRV36WT3M3K5LIJB3LCMLG6OFPQWWOMYHFOHGKLRECB7NE5VBEGA, seg_pos={0 0}, upload=102799]: Long tail cancellation due to optimal threshold Dec 10 13:15:37.200156243 (117h46m37.071239288s) [noise=true , address=145.79.227.40:4049 , piece_id=FVAQGZN6T3VF2XVZB4PCEYGS3PU3XGLBSN4FLFQTYIEC5HDTC3OQ, seg_pos={0 0}, upload=102799]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J): protocol: write tcp 188.166.225.170:55280->145.79.227.40:4049: use of closed network connection; write tcp 188.166.225.170:55280->145.79.227.40:4049: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55280->145.79.227.40:4049: use of closed network connection Dec 10 13:15:37.200184998 (117h46m37.071268053s) [noise=true , address=145.79.227.40:4049 , piece_id=FVAQGZN6T3VF2XVZB4PCEYGS3PU3XGLBSN4FLFQTYIEC5HDTC3OQ, seg_pos={0 0}, upload=102799]: Long tail cancellation due to optimal threshold Dec 10 13:15:37.200373426 (117h46m37.071456461s) [noise=true , address=45.91.116.11:4119 , piece_id=S5KLL473HCDKQBM5NVTVNLLSYGMXFFTFDFQX7D5SITM2MQO43K6Q, seg_pos={0 0}, upload=102799]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: expected piece hash; context canceled; EOF Dec 10 13:15:37.200381421 (117h46m37.071464466s) [noise=true , address=45.91.116.11:4119 , piece_id=S5KLL473HCDKQBM5NVTVNLLSYGMXFFTFDFQX7D5SITM2MQO43K6Q, seg_pos={0 0}, upload=102799]: Long tail cancellation due to optimal threshold Dec 10 13:15:37.200402562 (117h46m37.071485597s) [seg_pos={0 0}, upload=102799]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 10 13:15:37.200491951 (117h46m37.071574996s) [upload=102799]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 10 13:15:37.200500108 (117h46m37.071583153s) [upload=102799]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 10 13:15:37.624562388 (117h46m37.495645534s) [upload=102799]: Done uploading segments... Dec 10 13:15:37.624584852 (117h46m37.495667887s) [upload=102799]: Upload finished. err: <nil> Duration: 14.065528828s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 55 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXX..X.XXXX.XXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXX] duration: 5.002148128s trace: https://jaeger.dev.storj.io/trace/1638cad249a540f3 FAIL (2025-12-10 13:33:34 UTC) orbiter warning: context deadline exceeded Duration: 5.002173405s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [X.................................X.........................] duration: 24.126807337s trace: https://jaeger.dev.storj.io/trace/19144dc231ae8183 FAIL (2025-12-10 13:33:20 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ts... Dec 10 13:15:32.674883550 (117h46m32.545966595s) [upload=102799]: Got next segment. Inline: false Dec 10 13:15:32.674940880 (117h46m32.546023915s) [upload=102799]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 10 13:15:32.674946670 (117h46m32.546029706s) [upload=102799]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 10 13:15:33.036416584 (117h46m32.907499629s) [seg_pos={0 0}, upload=102799]: Begin upload segment... Dec 10 13:15:33.036472831 (117h46m32.907555876s) [seg_pos={0 0}, upload=102799]: Done begin upload segment. Dec 10 13:15:33.036565578 (117h46m32.907648613s) [noise=true , address=145.79.227.29:4943 , piece_id=I5NHKVGJKAG37VAV53QOQ2ZYSDF6SGEXTTG7DEAAG5CC2DO7SVVQ, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.036606606 (117h46m32.907689651s) [noise=true , address=145.79.227.6:4259 , piece_id=RQCMHQGVQGAWFBJ6AAHMVON7SLBV7XPYR4D6XRI6DJEJVJ7AFGRQ, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.037385819 (117h46m32.908468865s) [noise=true , address=145.79.227.26:4654 , piece_id=TRV36WT3M3K5LIJB3LCMLG6OFPQWWOMYHFOHGKLRECB7NE5VBEGA, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.037833743 (117h46m32.908916778s) [noise=true , address=145.79.227.22:4509 , piece_id=62EZ3JKY7O5VZEGSATQYYBLKR46ASNJUX3H7YIYAIXAV4WYF26JQ, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.038339726 (117h46m32.909422772s) [noise=true , address=145.79.227.10:4401 , piece_id=S4DH6HYJ5IUWJ5BTURVW7OTDWFS3O3UNCZMSTUK4QZ4OCNKTH6MA, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.038758464 (117h46m32.909841509s) [noise=true , address=145.79.227.40:4049 , piece_id=FVAQGZN6T3VF2XVZB4PCEYGS3PU3XGLBSN4FLFQTYIEC5HDTC3OQ, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.039163404 (117h46m32.910246449s) [noise=true , address=77.90.31.17:4771 , piece_id=X57ZVYWUTGUUKAOIRL2FL5CHXAN5SQYLX6EJBQIKL4FZGHWGPRQA, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.039588294 (117h46m32.910671339s) [noise=true , address=145.79.227.39:4014 , piece_id=HO5AOQJRC774MOI5S6J7MOHIYV4L72XC22HLQCHBLYP3QYIA2DTQ, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.040016559 (117h46m32.911099594s) [noise=true , address=145.79.227.43:4159 , piece_id=5KI6NRQSBBPKKUN65NU5LTY32YJ65K2OKZDJQEVQX465WNQKNHHQ, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.040450956 (117h46m32.911534001s) [noise=true , address=77.90.31.10:4518 , piece_id=XCU5D2A36LBTDDYTILXWUMMANHVDCW2KMEAB3HUCDU5UONZLULBQ, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.040890654 (117h46m32.911973689s) [noise=true , address=145.79.227.22:4510 , piece_id=ALCQVB6CUD7JAJ7I2NHADFKGHOSFVBKCGR2KHZWHPMYKK3VV4T5A, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.041348756 (117h46m32.912431801s) [noise=true , address=45.91.116.11:4119 , piece_id=S5KLL473HCDKQBM5NVTVNLLSYGMXFFTFDFQX7D5SITM2MQO43K6Q, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.041761201 (117h46m32.912844236s) [noise=true , address=145.79.227.41:4087 , piece_id=OQCN6TONOCOR2EH7THCYREB7LRVYOMK55VPTAWK4NMADKYYYPYLA, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.042078305 (117h46m32.91316134s) [noise=true , address=145.79.227.7:4295 , piece_id=IPMI23BRBMF4MTC6EJJEZWGW7AMFPLGUGMXEFCF2QZASWEUFIH5Q, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.042443110 (117h46m32.913526155s) [noise=true , address=145.79.227.32:4835 , piece_id=4X2XQSZAB6KZPDY5G4VFDFZCL744RAL2IYUYMU7UWFMEII5D225A, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.042561475 (117h46m32.91364452s) [noise=true , address=145.79.227.31:4797 , piece_id=WOGJMBDWY42R5T5T5M2NR4AF3NFZ6CAYOI2O5RRL3JYPCAB3FFZA, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:35.033962808 (117h46m34.905045853s) [upload=102799]: Next returned nil segment Dec 10 13:15:35.033993557 (117h46m34.905076612s) [upload=102799]: Waiting for error group managing segments... Dec 10 13:15:35.355027663 (117h46m35.226110708s) [noise=true , address=77.90.31.10:4518 , piece_id=XCU5D2A36LBTDDYTILXWUMMANHVDCW2KMEAB3HUCDU5UONZLULBQ, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:35.395849857 (117h46m35.266932902s) [noise=true , address=77.90.31.17:4771 , piece_id=X57ZVYWUTGUUKAOIRL2FL5CHXAN5SQYLX6EJBQIKL4FZGHWGPRQA, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:35.939806675 (117h46m35.810889721s) [noise=true , address=145.79.227.31:4797 , piece_id=WOGJMBDWY42R5T5T5M2NR4AF3NFZ6CAYOI2O5RRL3JYPCAB3FFZA, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:35.944824321 (117h46m35.815907356s) [noise=true , address=145.79.227.7:4295 , piece_id=IPMI23BRBMF4MTC6EJJEZWGW7AMFPLGUGMXEFCF2QZASWEUFIH5Q, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:35.966925235 (117h46m35.83800827s) [noise=true , address=145.79.227.41:4087 , piece_id=OQCN6TONOCOR2EH7THCYREB7LRVYOMK55VPTAWK4NMADKYYYPYLA, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:36.183487725 (117h46m36.05457076s) [noise=true , address=145.79.227.29:4943 , piece_id=I5NHKVGJKAG37VAV53QOQ2ZYSDF6SGEXTTG7DEAAG5CC2DO7SVVQ, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:36.184533436 (117h46m36.055616471s) [noise=true , address=145.79.227.22:4509 , piece_id=62EZ3JKY7O5VZEGSATQYYBLKR46ASNJUX3H7YIYAIXAV4WYF26JQ, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:36.188784180 (117h46m36.059867215s) [noise=true , address=145.79.227.6:4259 , piece_id=RQCMHQGVQGAWFBJ6AAHMVON7SLBV7XPYR4D6XRI6DJEJVJ7AFGRQ, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:36.190871565 (117h46m36.0619546s) [noise=true , address=145.79.227.22:4510 , piece_id=ALCQVB6CUD7JAJ7I2NHADFKGHOSFVBKCGR2KHZWHPMYKK3VV4T5A, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:36.234924680 (117h46m36.106007725s) [noise=true , address=145.79.227.39:4014 , piece_id=HO5AOQJRC774MOI5S6J7MOHIYV4L72XC22HLQCHBLYP3QYIA2DTQ, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:37.136102891 (117h46m37.007185926s) [noise=true , address=145.79.227.32:4835 , piece_id=4X2XQSZAB6KZPDY5G4VFDFZCL744RAL2IYUYMU7UWFMEII5D225A, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:37.155928563 (117h46m37.027011618s) [noise=true , address=145.79.227.43:4159 , piece_id=5KI6NRQSBBPKKUN65NU5LTY32YJ65K2OKZDJQEVQX465WNQKNHHQ, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:37.199763735 (117h46m37.0708468s) [noise=true , address=145.79.227.10:4401 , piece_id=S4DH6HYJ5IUWJ5BTURVW7OTDWFS3O3UNCZMSTUK4QZ4OCNKTH6MA, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:37.199789233 (117h46m37.070872279s) [seg_pos={0 0}, upload=102799]: Segment reached optimal threshold of 13 pieces. Dec 10 13:15:37.200041654 (117h46m37.071124689s) [noise=true , address=145.79.227.26:4654 , piece_id=TRV36WT3M3K5LIJB3LCMLG6OFPQWWOMYHFOHGKLRECB7NE5VBEGA, seg_pos={0 0}, upload=102799]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KhBWMerZe9tUmzdVEYkBb6NmJ25dGP26Ea7CLjNJT5gdAeiah): protocol: write tcp 188.166.225.170:53216->145.79.227.26:4654: use of closed network connection; write tcp 188.166.225.170:53216->145.79.227.26:4654: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53216->145.79.227.26:4654: use of closed network connection Dec 10 13:15:37.200055591 (117h46m37.071138656s) [noise=true , address=145.79.227.26:4654 , piece_id=TRV36WT3M3K5LIJB3LCMLG6OFPQWWOMYHFOHGKLRECB7NE5VBEGA, seg_pos={0 0}, upload=102799]: Long tail cancellation due to optimal threshold Dec 10 13:15:37.200156243 (117h46m37.071239288s) [noise=true , address=145.79.227.40:4049 , piece_id=FVAQGZN6T3VF2XVZB4PCEYGS3PU3XGLBSN4FLFQTYIEC5HDTC3OQ, seg_pos={0 0}, upload=102799]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J): protocol: write tcp 188.166.225.170:55280->145.79.227.40:4049: use of closed network connection; write tcp 188.166.225.170:55280->145.79.227.40:4049: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55280->145.79.227.40:4049: use of closed network connection Dec 10 13:15:37.200184998 (117h46m37.071268053s) [noise=true , address=145.79.227.40:4049 , piece_id=FVAQGZN6T3VF2XVZB4PCEYGS3PU3XGLBSN4FLFQTYIEC5HDTC3OQ, seg_pos={0 0}, upload=102799]: Long tail cancellation due to optimal threshold Dec 10 13:15:37.200373426 (117h46m37.071456461s) [noise=true , address=45.91.116.11:4119 , piece_id=S5KLL473HCDKQBM5NVTVNLLSYGMXFFTFDFQX7D5SITM2MQO43K6Q, seg_pos={0 0}, upload=102799]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: expected piece hash; context canceled; EOF Dec 10 13:15:37.200381421 (117h46m37.071464466s) [noise=true , address=45.91.116.11:4119 , piece_id=S5KLL473HCDKQBM5NVTVNLLSYGMXFFTFDFQX7D5SITM2MQO43K6Q, seg_pos={0 0}, upload=102799]: Long tail cancellation due to optimal threshold Dec 10 13:15:37.200402562 (117h46m37.071485597s) [seg_pos={0 0}, upload=102799]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 10 13:15:37.200491951 (117h46m37.071574996s) [upload=102799]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 10 13:15:37.200500108 (117h46m37.071583153s) [upload=102799]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 10 13:15:37.624562388 (117h46m37.495645534s) [upload=102799]: Done uploading segments... Dec 10 13:15:37.624584852 (117h46m37.495667887s) [upload=102799]: Upload finished. err: <nil> Duration: 24.126849036s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 3.770158545s trace: https://jaeger.dev.storj.io/trace/a0b5a94505b001 FAIL (2025-12-10 13:33:10 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:36068->45.38.24.2:443: read: connection reset by peer Duration: 3.770175847s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.X...........................................X..X.X....X....] duration: 11.770774581s trace: https://jaeger.dev.storj.io/trace/70ffd77b4c08e04c FAIL (2025-12-10 13:32:56 UTC) orbiter warning: context deadline exceeded Duration: 11.770798627s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.X.................................X........................] duration: 17.108376775s trace: https://jaeger.dev.storj.io/trace/19144dc231ae8183 FAIL (2025-12-10 13:32:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ts... Dec 10 13:15:32.674883550 (117h46m32.545966595s) [upload=102799]: Got next segment. Inline: false Dec 10 13:15:32.674940880 (117h46m32.546023915s) [upload=102799]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 10 13:15:32.674946670 (117h46m32.546029706s) [upload=102799]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 10 13:15:33.036416584 (117h46m32.907499629s) [seg_pos={0 0}, upload=102799]: Begin upload segment... Dec 10 13:15:33.036472831 (117h46m32.907555876s) [seg_pos={0 0}, upload=102799]: Done begin upload segment. Dec 10 13:15:33.036565578 (117h46m32.907648613s) [noise=true , address=145.79.227.29:4943 , piece_id=I5NHKVGJKAG37VAV53QOQ2ZYSDF6SGEXTTG7DEAAG5CC2DO7SVVQ, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.036606606 (117h46m32.907689651s) [noise=true , address=145.79.227.6:4259 , piece_id=RQCMHQGVQGAWFBJ6AAHMVON7SLBV7XPYR4D6XRI6DJEJVJ7AFGRQ, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.037385819 (117h46m32.908468865s) [noise=true , address=145.79.227.26:4654 , piece_id=TRV36WT3M3K5LIJB3LCMLG6OFPQWWOMYHFOHGKLRECB7NE5VBEGA, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.037833743 (117h46m32.908916778s) [noise=true , address=145.79.227.22:4509 , piece_id=62EZ3JKY7O5VZEGSATQYYBLKR46ASNJUX3H7YIYAIXAV4WYF26JQ, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.038339726 (117h46m32.909422772s) [noise=true , address=145.79.227.10:4401 , piece_id=S4DH6HYJ5IUWJ5BTURVW7OTDWFS3O3UNCZMSTUK4QZ4OCNKTH6MA, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.038758464 (117h46m32.909841509s) [noise=true , address=145.79.227.40:4049 , piece_id=FVAQGZN6T3VF2XVZB4PCEYGS3PU3XGLBSN4FLFQTYIEC5HDTC3OQ, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.039163404 (117h46m32.910246449s) [noise=true , address=77.90.31.17:4771 , piece_id=X57ZVYWUTGUUKAOIRL2FL5CHXAN5SQYLX6EJBQIKL4FZGHWGPRQA, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.039588294 (117h46m32.910671339s) [noise=true , address=145.79.227.39:4014 , piece_id=HO5AOQJRC774MOI5S6J7MOHIYV4L72XC22HLQCHBLYP3QYIA2DTQ, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.040016559 (117h46m32.911099594s) [noise=true , address=145.79.227.43:4159 , piece_id=5KI6NRQSBBPKKUN65NU5LTY32YJ65K2OKZDJQEVQX465WNQKNHHQ, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.040450956 (117h46m32.911534001s) [noise=true , address=77.90.31.10:4518 , piece_id=XCU5D2A36LBTDDYTILXWUMMANHVDCW2KMEAB3HUCDU5UONZLULBQ, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.040890654 (117h46m32.911973689s) [noise=true , address=145.79.227.22:4510 , piece_id=ALCQVB6CUD7JAJ7I2NHADFKGHOSFVBKCGR2KHZWHPMYKK3VV4T5A, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.041348756 (117h46m32.912431801s) [noise=true , address=45.91.116.11:4119 , piece_id=S5KLL473HCDKQBM5NVTVNLLSYGMXFFTFDFQX7D5SITM2MQO43K6Q, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.041761201 (117h46m32.912844236s) [noise=true , address=145.79.227.41:4087 , piece_id=OQCN6TONOCOR2EH7THCYREB7LRVYOMK55VPTAWK4NMADKYYYPYLA, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.042078305 (117h46m32.91316134s) [noise=true , address=145.79.227.7:4295 , piece_id=IPMI23BRBMF4MTC6EJJEZWGW7AMFPLGUGMXEFCF2QZASWEUFIH5Q, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.042443110 (117h46m32.913526155s) [noise=true , address=145.79.227.32:4835 , piece_id=4X2XQSZAB6KZPDY5G4VFDFZCL744RAL2IYUYMU7UWFMEII5D225A, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.042561475 (117h46m32.91364452s) [noise=true , address=145.79.227.31:4797 , piece_id=WOGJMBDWY42R5T5T5M2NR4AF3NFZ6CAYOI2O5RRL3JYPCAB3FFZA, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:35.033962808 (117h46m34.905045853s) [upload=102799]: Next returned nil segment Dec 10 13:15:35.033993557 (117h46m34.905076612s) [upload=102799]: Waiting for error group managing segments... Dec 10 13:15:35.355027663 (117h46m35.226110708s) [noise=true , address=77.90.31.10:4518 , piece_id=XCU5D2A36LBTDDYTILXWUMMANHVDCW2KMEAB3HUCDU5UONZLULBQ, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:35.395849857 (117h46m35.266932902s) [noise=true , address=77.90.31.17:4771 , piece_id=X57ZVYWUTGUUKAOIRL2FL5CHXAN5SQYLX6EJBQIKL4FZGHWGPRQA, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:35.939806675 (117h46m35.810889721s) [noise=true , address=145.79.227.31:4797 , piece_id=WOGJMBDWY42R5T5T5M2NR4AF3NFZ6CAYOI2O5RRL3JYPCAB3FFZA, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:35.944824321 (117h46m35.815907356s) [noise=true , address=145.79.227.7:4295 , piece_id=IPMI23BRBMF4MTC6EJJEZWGW7AMFPLGUGMXEFCF2QZASWEUFIH5Q, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:35.966925235 (117h46m35.83800827s) [noise=true , address=145.79.227.41:4087 , piece_id=OQCN6TONOCOR2EH7THCYREB7LRVYOMK55VPTAWK4NMADKYYYPYLA, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:36.183487725 (117h46m36.05457076s) [noise=true , address=145.79.227.29:4943 , piece_id=I5NHKVGJKAG37VAV53QOQ2ZYSDF6SGEXTTG7DEAAG5CC2DO7SVVQ, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:36.184533436 (117h46m36.055616471s) [noise=true , address=145.79.227.22:4509 , piece_id=62EZ3JKY7O5VZEGSATQYYBLKR46ASNJUX3H7YIYAIXAV4WYF26JQ, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:36.188784180 (117h46m36.059867215s) [noise=true , address=145.79.227.6:4259 , piece_id=RQCMHQGVQGAWFBJ6AAHMVON7SLBV7XPYR4D6XRI6DJEJVJ7AFGRQ, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:36.190871565 (117h46m36.0619546s) [noise=true , address=145.79.227.22:4510 , piece_id=ALCQVB6CUD7JAJ7I2NHADFKGHOSFVBKCGR2KHZWHPMYKK3VV4T5A, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:36.234924680 (117h46m36.106007725s) [noise=true , address=145.79.227.39:4014 , piece_id=HO5AOQJRC774MOI5S6J7MOHIYV4L72XC22HLQCHBLYP3QYIA2DTQ, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:37.136102891 (117h46m37.007185926s) [noise=true , address=145.79.227.32:4835 , piece_id=4X2XQSZAB6KZPDY5G4VFDFZCL744RAL2IYUYMU7UWFMEII5D225A, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:37.155928563 (117h46m37.027011618s) [noise=true , address=145.79.227.43:4159 , piece_id=5KI6NRQSBBPKKUN65NU5LTY32YJ65K2OKZDJQEVQX465WNQKNHHQ, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:37.199763735 (117h46m37.0708468s) [noise=true , address=145.79.227.10:4401 , piece_id=S4DH6HYJ5IUWJ5BTURVW7OTDWFS3O3UNCZMSTUK4QZ4OCNKTH6MA, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:37.199789233 (117h46m37.070872279s) [seg_pos={0 0}, upload=102799]: Segment reached optimal threshold of 13 pieces. Dec 10 13:15:37.200041654 (117h46m37.071124689s) [noise=true , address=145.79.227.26:4654 , piece_id=TRV36WT3M3K5LIJB3LCMLG6OFPQWWOMYHFOHGKLRECB7NE5VBEGA, seg_pos={0 0}, upload=102799]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KhBWMerZe9tUmzdVEYkBb6NmJ25dGP26Ea7CLjNJT5gdAeiah): protocol: write tcp 188.166.225.170:53216->145.79.227.26:4654: use of closed network connection; write tcp 188.166.225.170:53216->145.79.227.26:4654: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53216->145.79.227.26:4654: use of closed network connection Dec 10 13:15:37.200055591 (117h46m37.071138656s) [noise=true , address=145.79.227.26:4654 , piece_id=TRV36WT3M3K5LIJB3LCMLG6OFPQWWOMYHFOHGKLRECB7NE5VBEGA, seg_pos={0 0}, upload=102799]: Long tail cancellation due to optimal threshold Dec 10 13:15:37.200156243 (117h46m37.071239288s) [noise=true , address=145.79.227.40:4049 , piece_id=FVAQGZN6T3VF2XVZB4PCEYGS3PU3XGLBSN4FLFQTYIEC5HDTC3OQ, seg_pos={0 0}, upload=102799]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J): protocol: write tcp 188.166.225.170:55280->145.79.227.40:4049: use of closed network connection; write tcp 188.166.225.170:55280->145.79.227.40:4049: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55280->145.79.227.40:4049: use of closed network connection Dec 10 13:15:37.200184998 (117h46m37.071268053s) [noise=true , address=145.79.227.40:4049 , piece_id=FVAQGZN6T3VF2XVZB4PCEYGS3PU3XGLBSN4FLFQTYIEC5HDTC3OQ, seg_pos={0 0}, upload=102799]: Long tail cancellation due to optimal threshold Dec 10 13:15:37.200373426 (117h46m37.071456461s) [noise=true , address=45.91.116.11:4119 , piece_id=S5KLL473HCDKQBM5NVTVNLLSYGMXFFTFDFQX7D5SITM2MQO43K6Q, seg_pos={0 0}, upload=102799]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: expected piece hash; context canceled; EOF Dec 10 13:15:37.200381421 (117h46m37.071464466s) [noise=true , address=45.91.116.11:4119 , piece_id=S5KLL473HCDKQBM5NVTVNLLSYGMXFFTFDFQX7D5SITM2MQO43K6Q, seg_pos={0 0}, upload=102799]: Long tail cancellation due to optimal threshold Dec 10 13:15:37.200402562 (117h46m37.071485597s) [seg_pos={0 0}, upload=102799]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 10 13:15:37.200491951 (117h46m37.071574996s) [upload=102799]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 10 13:15:37.200500108 (117h46m37.071583153s) [upload=102799]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 10 13:15:37.624562388 (117h46m37.495645534s) [upload=102799]: Done uploading segments... Dec 10 13:15:37.624584852 (117h46m37.495667887s) [upload=102799]: Upload finished. err: <nil> Duration: 17.108407883s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 55 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXX..X.XXXX.XXXXXXXXXXXX.XXXXXXXXXXXXXXXXXX] duration: 5.00081914s trace: https://jaeger.dev.storj.io/trace/22e90f5351f69a42 FAIL (2025-12-10 13:32:44 UTC) orbiter warning: context deadline exceeded Duration: 5.000842043s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..X.................................X.......................] duration: 9.66219931s trace: https://jaeger.dev.storj.io/trace/19144dc231ae8183 FAIL (2025-12-10 13:32:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ts... Dec 10 13:15:32.674883550 (117h46m32.545966595s) [upload=102799]: Got next segment. Inline: false Dec 10 13:15:32.674940880 (117h46m32.546023915s) [upload=102799]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 10 13:15:32.674946670 (117h46m32.546029706s) [upload=102799]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 10 13:15:33.036416584 (117h46m32.907499629s) [seg_pos={0 0}, upload=102799]: Begin upload segment... Dec 10 13:15:33.036472831 (117h46m32.907555876s) [seg_pos={0 0}, upload=102799]: Done begin upload segment. Dec 10 13:15:33.036565578 (117h46m32.907648613s) [noise=true , address=145.79.227.29:4943 , piece_id=I5NHKVGJKAG37VAV53QOQ2ZYSDF6SGEXTTG7DEAAG5CC2DO7SVVQ, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.036606606 (117h46m32.907689651s) [noise=true , address=145.79.227.6:4259 , piece_id=RQCMHQGVQGAWFBJ6AAHMVON7SLBV7XPYR4D6XRI6DJEJVJ7AFGRQ, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.037385819 (117h46m32.908468865s) [noise=true , address=145.79.227.26:4654 , piece_id=TRV36WT3M3K5LIJB3LCMLG6OFPQWWOMYHFOHGKLRECB7NE5VBEGA, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.037833743 (117h46m32.908916778s) [noise=true , address=145.79.227.22:4509 , piece_id=62EZ3JKY7O5VZEGSATQYYBLKR46ASNJUX3H7YIYAIXAV4WYF26JQ, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.038339726 (117h46m32.909422772s) [noise=true , address=145.79.227.10:4401 , piece_id=S4DH6HYJ5IUWJ5BTURVW7OTDWFS3O3UNCZMSTUK4QZ4OCNKTH6MA, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.038758464 (117h46m32.909841509s) [noise=true , address=145.79.227.40:4049 , piece_id=FVAQGZN6T3VF2XVZB4PCEYGS3PU3XGLBSN4FLFQTYIEC5HDTC3OQ, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.039163404 (117h46m32.910246449s) [noise=true , address=77.90.31.17:4771 , piece_id=X57ZVYWUTGUUKAOIRL2FL5CHXAN5SQYLX6EJBQIKL4FZGHWGPRQA, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.039588294 (117h46m32.910671339s) [noise=true , address=145.79.227.39:4014 , piece_id=HO5AOQJRC774MOI5S6J7MOHIYV4L72XC22HLQCHBLYP3QYIA2DTQ, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.040016559 (117h46m32.911099594s) [noise=true , address=145.79.227.43:4159 , piece_id=5KI6NRQSBBPKKUN65NU5LTY32YJ65K2OKZDJQEVQX465WNQKNHHQ, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.040450956 (117h46m32.911534001s) [noise=true , address=77.90.31.10:4518 , piece_id=XCU5D2A36LBTDDYTILXWUMMANHVDCW2KMEAB3HUCDU5UONZLULBQ, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.040890654 (117h46m32.911973689s) [noise=true , address=145.79.227.22:4510 , piece_id=ALCQVB6CUD7JAJ7I2NHADFKGHOSFVBKCGR2KHZWHPMYKK3VV4T5A, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.041348756 (117h46m32.912431801s) [noise=true , address=45.91.116.11:4119 , piece_id=S5KLL473HCDKQBM5NVTVNLLSYGMXFFTFDFQX7D5SITM2MQO43K6Q, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.041761201 (117h46m32.912844236s) [noise=true , address=145.79.227.41:4087 , piece_id=OQCN6TONOCOR2EH7THCYREB7LRVYOMK55VPTAWK4NMADKYYYPYLA, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.042078305 (117h46m32.91316134s) [noise=true , address=145.79.227.7:4295 , piece_id=IPMI23BRBMF4MTC6EJJEZWGW7AMFPLGUGMXEFCF2QZASWEUFIH5Q, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.042443110 (117h46m32.913526155s) [noise=true , address=145.79.227.32:4835 , piece_id=4X2XQSZAB6KZPDY5G4VFDFZCL744RAL2IYUYMU7UWFMEII5D225A, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.042561475 (117h46m32.91364452s) [noise=true , address=145.79.227.31:4797 , piece_id=WOGJMBDWY42R5T5T5M2NR4AF3NFZ6CAYOI2O5RRL3JYPCAB3FFZA, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:35.033962808 (117h46m34.905045853s) [upload=102799]: Next returned nil segment Dec 10 13:15:35.033993557 (117h46m34.905076612s) [upload=102799]: Waiting for error group managing segments... Dec 10 13:15:35.355027663 (117h46m35.226110708s) [noise=true , address=77.90.31.10:4518 , piece_id=XCU5D2A36LBTDDYTILXWUMMANHVDCW2KMEAB3HUCDU5UONZLULBQ, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:35.395849857 (117h46m35.266932902s) [noise=true , address=77.90.31.17:4771 , piece_id=X57ZVYWUTGUUKAOIRL2FL5CHXAN5SQYLX6EJBQIKL4FZGHWGPRQA, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:35.939806675 (117h46m35.810889721s) [noise=true , address=145.79.227.31:4797 , piece_id=WOGJMBDWY42R5T5T5M2NR4AF3NFZ6CAYOI2O5RRL3JYPCAB3FFZA, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:35.944824321 (117h46m35.815907356s) [noise=true , address=145.79.227.7:4295 , piece_id=IPMI23BRBMF4MTC6EJJEZWGW7AMFPLGUGMXEFCF2QZASWEUFIH5Q, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:35.966925235 (117h46m35.83800827s) [noise=true , address=145.79.227.41:4087 , piece_id=OQCN6TONOCOR2EH7THCYREB7LRVYOMK55VPTAWK4NMADKYYYPYLA, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:36.183487725 (117h46m36.05457076s) [noise=true , address=145.79.227.29:4943 , piece_id=I5NHKVGJKAG37VAV53QOQ2ZYSDF6SGEXTTG7DEAAG5CC2DO7SVVQ, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:36.184533436 (117h46m36.055616471s) [noise=true , address=145.79.227.22:4509 , piece_id=62EZ3JKY7O5VZEGSATQYYBLKR46ASNJUX3H7YIYAIXAV4WYF26JQ, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:36.188784180 (117h46m36.059867215s) [noise=true , address=145.79.227.6:4259 , piece_id=RQCMHQGVQGAWFBJ6AAHMVON7SLBV7XPYR4D6XRI6DJEJVJ7AFGRQ, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:36.190871565 (117h46m36.0619546s) [noise=true , address=145.79.227.22:4510 , piece_id=ALCQVB6CUD7JAJ7I2NHADFKGHOSFVBKCGR2KHZWHPMYKK3VV4T5A, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:36.234924680 (117h46m36.106007725s) [noise=true , address=145.79.227.39:4014 , piece_id=HO5AOQJRC774MOI5S6J7MOHIYV4L72XC22HLQCHBLYP3QYIA2DTQ, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:37.136102891 (117h46m37.007185926s) [noise=true , address=145.79.227.32:4835 , piece_id=4X2XQSZAB6KZPDY5G4VFDFZCL744RAL2IYUYMU7UWFMEII5D225A, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:37.155928563 (117h46m37.027011618s) [noise=true , address=145.79.227.43:4159 , piece_id=5KI6NRQSBBPKKUN65NU5LTY32YJ65K2OKZDJQEVQX465WNQKNHHQ, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:37.199763735 (117h46m37.0708468s) [noise=true , address=145.79.227.10:4401 , piece_id=S4DH6HYJ5IUWJ5BTURVW7OTDWFS3O3UNCZMSTUK4QZ4OCNKTH6MA, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:37.199789233 (117h46m37.070872279s) [seg_pos={0 0}, upload=102799]: Segment reached optimal threshold of 13 pieces. Dec 10 13:15:37.200041654 (117h46m37.071124689s) [noise=true , address=145.79.227.26:4654 , piece_id=TRV36WT3M3K5LIJB3LCMLG6OFPQWWOMYHFOHGKLRECB7NE5VBEGA, seg_pos={0 0}, upload=102799]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KhBWMerZe9tUmzdVEYkBb6NmJ25dGP26Ea7CLjNJT5gdAeiah): protocol: write tcp 188.166.225.170:53216->145.79.227.26:4654: use of closed network connection; write tcp 188.166.225.170:53216->145.79.227.26:4654: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53216->145.79.227.26:4654: use of closed network connection Dec 10 13:15:37.200055591 (117h46m37.071138656s) [noise=true , address=145.79.227.26:4654 , piece_id=TRV36WT3M3K5LIJB3LCMLG6OFPQWWOMYHFOHGKLRECB7NE5VBEGA, seg_pos={0 0}, upload=102799]: Long tail cancellation due to optimal threshold Dec 10 13:15:37.200156243 (117h46m37.071239288s) [noise=true , address=145.79.227.40:4049 , piece_id=FVAQGZN6T3VF2XVZB4PCEYGS3PU3XGLBSN4FLFQTYIEC5HDTC3OQ, seg_pos={0 0}, upload=102799]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J): protocol: write tcp 188.166.225.170:55280->145.79.227.40:4049: use of closed network connection; write tcp 188.166.225.170:55280->145.79.227.40:4049: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55280->145.79.227.40:4049: use of closed network connection Dec 10 13:15:37.200184998 (117h46m37.071268053s) [noise=true , address=145.79.227.40:4049 , piece_id=FVAQGZN6T3VF2XVZB4PCEYGS3PU3XGLBSN4FLFQTYIEC5HDTC3OQ, seg_pos={0 0}, upload=102799]: Long tail cancellation due to optimal threshold Dec 10 13:15:37.200373426 (117h46m37.071456461s) [noise=true , address=45.91.116.11:4119 , piece_id=S5KLL473HCDKQBM5NVTVNLLSYGMXFFTFDFQX7D5SITM2MQO43K6Q, seg_pos={0 0}, upload=102799]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: expected piece hash; context canceled; EOF Dec 10 13:15:37.200381421 (117h46m37.071464466s) [noise=true , address=45.91.116.11:4119 , piece_id=S5KLL473HCDKQBM5NVTVNLLSYGMXFFTFDFQX7D5SITM2MQO43K6Q, seg_pos={0 0}, upload=102799]: Long tail cancellation due to optimal threshold Dec 10 13:15:37.200402562 (117h46m37.071485597s) [seg_pos={0 0}, upload=102799]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 10 13:15:37.200491951 (117h46m37.071574996s) [upload=102799]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 10 13:15:37.200500108 (117h46m37.071583153s) [upload=102799]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 10 13:15:37.624562388 (117h46m37.495645534s) [upload=102799]: Done uploading segments... Dec 10 13:15:37.624584852 (117h46m37.495667887s) [upload=102799]: Upload finished. err: <nil> Duration: 9.662229839s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [..X...........................................X..X.X....X...] duration: 6.680564808s trace: https://jaeger.dev.storj.io/trace/70ffd77b4c08e04c FAIL (2025-12-10 13:32:12 UTC) orbiter warning: context deadline exceeded Duration: 6.6806344s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 3.205918065s trace: https://jaeger.dev.storj.io/trace/a0b5a94505b001 FAIL (2025-12-10 13:32:12 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:36068->45.38.24.2:443: read: connection reset by peer Duration: 3.205947263s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...X.................................X......................] duration: 13.456328095s trace: https://jaeger.dev.storj.io/trace/19144dc231ae8183 FAIL (2025-12-10 13:31:47 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ts... Dec 10 13:15:32.674883550 (117h46m32.545966595s) [upload=102799]: Got next segment. Inline: false Dec 10 13:15:32.674940880 (117h46m32.546023915s) [upload=102799]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 10 13:15:32.674946670 (117h46m32.546029706s) [upload=102799]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 10 13:15:33.036416584 (117h46m32.907499629s) [seg_pos={0 0}, upload=102799]: Begin upload segment... Dec 10 13:15:33.036472831 (117h46m32.907555876s) [seg_pos={0 0}, upload=102799]: Done begin upload segment. Dec 10 13:15:33.036565578 (117h46m32.907648613s) [noise=true , address=145.79.227.29:4943 , piece_id=I5NHKVGJKAG37VAV53QOQ2ZYSDF6SGEXTTG7DEAAG5CC2DO7SVVQ, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.036606606 (117h46m32.907689651s) [noise=true , address=145.79.227.6:4259 , piece_id=RQCMHQGVQGAWFBJ6AAHMVON7SLBV7XPYR4D6XRI6DJEJVJ7AFGRQ, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.037385819 (117h46m32.908468865s) [noise=true , address=145.79.227.26:4654 , piece_id=TRV36WT3M3K5LIJB3LCMLG6OFPQWWOMYHFOHGKLRECB7NE5VBEGA, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.037833743 (117h46m32.908916778s) [noise=true , address=145.79.227.22:4509 , piece_id=62EZ3JKY7O5VZEGSATQYYBLKR46ASNJUX3H7YIYAIXAV4WYF26JQ, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.038339726 (117h46m32.909422772s) [noise=true , address=145.79.227.10:4401 , piece_id=S4DH6HYJ5IUWJ5BTURVW7OTDWFS3O3UNCZMSTUK4QZ4OCNKTH6MA, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.038758464 (117h46m32.909841509s) [noise=true , address=145.79.227.40:4049 , piece_id=FVAQGZN6T3VF2XVZB4PCEYGS3PU3XGLBSN4FLFQTYIEC5HDTC3OQ, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.039163404 (117h46m32.910246449s) [noise=true , address=77.90.31.17:4771 , piece_id=X57ZVYWUTGUUKAOIRL2FL5CHXAN5SQYLX6EJBQIKL4FZGHWGPRQA, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.039588294 (117h46m32.910671339s) [noise=true , address=145.79.227.39:4014 , piece_id=HO5AOQJRC774MOI5S6J7MOHIYV4L72XC22HLQCHBLYP3QYIA2DTQ, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.040016559 (117h46m32.911099594s) [noise=true , address=145.79.227.43:4159 , piece_id=5KI6NRQSBBPKKUN65NU5LTY32YJ65K2OKZDJQEVQX465WNQKNHHQ, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.040450956 (117h46m32.911534001s) [noise=true , address=77.90.31.10:4518 , piece_id=XCU5D2A36LBTDDYTILXWUMMANHVDCW2KMEAB3HUCDU5UONZLULBQ, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.040890654 (117h46m32.911973689s) [noise=true , address=145.79.227.22:4510 , piece_id=ALCQVB6CUD7JAJ7I2NHADFKGHOSFVBKCGR2KHZWHPMYKK3VV4T5A, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.041348756 (117h46m32.912431801s) [noise=true , address=45.91.116.11:4119 , piece_id=S5KLL473HCDKQBM5NVTVNLLSYGMXFFTFDFQX7D5SITM2MQO43K6Q, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.041761201 (117h46m32.912844236s) [noise=true , address=145.79.227.41:4087 , piece_id=OQCN6TONOCOR2EH7THCYREB7LRVYOMK55VPTAWK4NMADKYYYPYLA, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.042078305 (117h46m32.91316134s) [noise=true , address=145.79.227.7:4295 , piece_id=IPMI23BRBMF4MTC6EJJEZWGW7AMFPLGUGMXEFCF2QZASWEUFIH5Q, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.042443110 (117h46m32.913526155s) [noise=true , address=145.79.227.32:4835 , piece_id=4X2XQSZAB6KZPDY5G4VFDFZCL744RAL2IYUYMU7UWFMEII5D225A, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.042561475 (117h46m32.91364452s) [noise=true , address=145.79.227.31:4797 , piece_id=WOGJMBDWY42R5T5T5M2NR4AF3NFZ6CAYOI2O5RRL3JYPCAB3FFZA, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:35.033962808 (117h46m34.905045853s) [upload=102799]: Next returned nil segment Dec 10 13:15:35.033993557 (117h46m34.905076612s) [upload=102799]: Waiting for error group managing segments... Dec 10 13:15:35.355027663 (117h46m35.226110708s) [noise=true , address=77.90.31.10:4518 , piece_id=XCU5D2A36LBTDDYTILXWUMMANHVDCW2KMEAB3HUCDU5UONZLULBQ, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:35.395849857 (117h46m35.266932902s) [noise=true , address=77.90.31.17:4771 , piece_id=X57ZVYWUTGUUKAOIRL2FL5CHXAN5SQYLX6EJBQIKL4FZGHWGPRQA, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:35.939806675 (117h46m35.810889721s) [noise=true , address=145.79.227.31:4797 , piece_id=WOGJMBDWY42R5T5T5M2NR4AF3NFZ6CAYOI2O5RRL3JYPCAB3FFZA, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:35.944824321 (117h46m35.815907356s) [noise=true , address=145.79.227.7:4295 , piece_id=IPMI23BRBMF4MTC6EJJEZWGW7AMFPLGUGMXEFCF2QZASWEUFIH5Q, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:35.966925235 (117h46m35.83800827s) [noise=true , address=145.79.227.41:4087 , piece_id=OQCN6TONOCOR2EH7THCYREB7LRVYOMK55VPTAWK4NMADKYYYPYLA, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:36.183487725 (117h46m36.05457076s) [noise=true , address=145.79.227.29:4943 , piece_id=I5NHKVGJKAG37VAV53QOQ2ZYSDF6SGEXTTG7DEAAG5CC2DO7SVVQ, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:36.184533436 (117h46m36.055616471s) [noise=true , address=145.79.227.22:4509 , piece_id=62EZ3JKY7O5VZEGSATQYYBLKR46ASNJUX3H7YIYAIXAV4WYF26JQ, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:36.188784180 (117h46m36.059867215s) [noise=true , address=145.79.227.6:4259 , piece_id=RQCMHQGVQGAWFBJ6AAHMVON7SLBV7XPYR4D6XRI6DJEJVJ7AFGRQ, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:36.190871565 (117h46m36.0619546s) [noise=true , address=145.79.227.22:4510 , piece_id=ALCQVB6CUD7JAJ7I2NHADFKGHOSFVBKCGR2KHZWHPMYKK3VV4T5A, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:36.234924680 (117h46m36.106007725s) [noise=true , address=145.79.227.39:4014 , piece_id=HO5AOQJRC774MOI5S6J7MOHIYV4L72XC22HLQCHBLYP3QYIA2DTQ, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:37.136102891 (117h46m37.007185926s) [noise=true , address=145.79.227.32:4835 , piece_id=4X2XQSZAB6KZPDY5G4VFDFZCL744RAL2IYUYMU7UWFMEII5D225A, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:37.155928563 (117h46m37.027011618s) [noise=true , address=145.79.227.43:4159 , piece_id=5KI6NRQSBBPKKUN65NU5LTY32YJ65K2OKZDJQEVQX465WNQKNHHQ, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:37.199763735 (117h46m37.0708468s) [noise=true , address=145.79.227.10:4401 , piece_id=S4DH6HYJ5IUWJ5BTURVW7OTDWFS3O3UNCZMSTUK4QZ4OCNKTH6MA, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:37.199789233 (117h46m37.070872279s) [seg_pos={0 0}, upload=102799]: Segment reached optimal threshold of 13 pieces. Dec 10 13:15:37.200041654 (117h46m37.071124689s) [noise=true , address=145.79.227.26:4654 , piece_id=TRV36WT3M3K5LIJB3LCMLG6OFPQWWOMYHFOHGKLRECB7NE5VBEGA, seg_pos={0 0}, upload=102799]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KhBWMerZe9tUmzdVEYkBb6NmJ25dGP26Ea7CLjNJT5gdAeiah): protocol: write tcp 188.166.225.170:53216->145.79.227.26:4654: use of closed network connection; write tcp 188.166.225.170:53216->145.79.227.26:4654: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53216->145.79.227.26:4654: use of closed network connection Dec 10 13:15:37.200055591 (117h46m37.071138656s) [noise=true , address=145.79.227.26:4654 , piece_id=TRV36WT3M3K5LIJB3LCMLG6OFPQWWOMYHFOHGKLRECB7NE5VBEGA, seg_pos={0 0}, upload=102799]: Long tail cancellation due to optimal threshold Dec 10 13:15:37.200156243 (117h46m37.071239288s) [noise=true , address=145.79.227.40:4049 , piece_id=FVAQGZN6T3VF2XVZB4PCEYGS3PU3XGLBSN4FLFQTYIEC5HDTC3OQ, seg_pos={0 0}, upload=102799]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J): protocol: write tcp 188.166.225.170:55280->145.79.227.40:4049: use of closed network connection; write tcp 188.166.225.170:55280->145.79.227.40:4049: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55280->145.79.227.40:4049: use of closed network connection Dec 10 13:15:37.200184998 (117h46m37.071268053s) [noise=true , address=145.79.227.40:4049 , piece_id=FVAQGZN6T3VF2XVZB4PCEYGS3PU3XGLBSN4FLFQTYIEC5HDTC3OQ, seg_pos={0 0}, upload=102799]: Long tail cancellation due to optimal threshold Dec 10 13:15:37.200373426 (117h46m37.071456461s) [noise=true , address=45.91.116.11:4119 , piece_id=S5KLL473HCDKQBM5NVTVNLLSYGMXFFTFDFQX7D5SITM2MQO43K6Q, seg_pos={0 0}, upload=102799]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: expected piece hash; context canceled; EOF Dec 10 13:15:37.200381421 (117h46m37.071464466s) [noise=true , address=45.91.116.11:4119 , piece_id=S5KLL473HCDKQBM5NVTVNLLSYGMXFFTFDFQX7D5SITM2MQO43K6Q, seg_pos={0 0}, upload=102799]: Long tail cancellation due to optimal threshold Dec 10 13:15:37.200402562 (117h46m37.071485597s) [seg_pos={0 0}, upload=102799]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 10 13:15:37.200491951 (117h46m37.071574996s) [upload=102799]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 10 13:15:37.200500108 (117h46m37.071583153s) [upload=102799]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 10 13:15:37.624562388 (117h46m37.495645534s) [upload=102799]: Done uploading segments... Dec 10 13:15:37.624584852 (117h46m37.495667887s) [upload=102799]: Upload finished. err: <nil> Duration: 13.456343694s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 8.805085721s trace: https://jaeger.dev.storj.io/trace/a0b5a94505b001 FAIL (2025-12-10 13:31:41 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:36068->45.38.24.2:443: read: connection reset by peer Duration: 8.805113242s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [X...X.................................X.....................] duration: 27.677353904s trace: https://jaeger.dev.storj.io/trace/19144dc231ae8183 FAIL (2025-12-10 13:31:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ts... Dec 10 13:15:32.674883550 (117h46m32.545966595s) [upload=102799]: Got next segment. Inline: false Dec 10 13:15:32.674940880 (117h46m32.546023915s) [upload=102799]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 10 13:15:32.674946670 (117h46m32.546029706s) [upload=102799]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 10 13:15:33.036416584 (117h46m32.907499629s) [seg_pos={0 0}, upload=102799]: Begin upload segment... Dec 10 13:15:33.036472831 (117h46m32.907555876s) [seg_pos={0 0}, upload=102799]: Done begin upload segment. Dec 10 13:15:33.036565578 (117h46m32.907648613s) [noise=true , address=145.79.227.29:4943 , piece_id=I5NHKVGJKAG37VAV53QOQ2ZYSDF6SGEXTTG7DEAAG5CC2DO7SVVQ, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.036606606 (117h46m32.907689651s) [noise=true , address=145.79.227.6:4259 , piece_id=RQCMHQGVQGAWFBJ6AAHMVON7SLBV7XPYR4D6XRI6DJEJVJ7AFGRQ, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.037385819 (117h46m32.908468865s) [noise=true , address=145.79.227.26:4654 , piece_id=TRV36WT3M3K5LIJB3LCMLG6OFPQWWOMYHFOHGKLRECB7NE5VBEGA, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.037833743 (117h46m32.908916778s) [noise=true , address=145.79.227.22:4509 , piece_id=62EZ3JKY7O5VZEGSATQYYBLKR46ASNJUX3H7YIYAIXAV4WYF26JQ, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.038339726 (117h46m32.909422772s) [noise=true , address=145.79.227.10:4401 , piece_id=S4DH6HYJ5IUWJ5BTURVW7OTDWFS3O3UNCZMSTUK4QZ4OCNKTH6MA, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.038758464 (117h46m32.909841509s) [noise=true , address=145.79.227.40:4049 , piece_id=FVAQGZN6T3VF2XVZB4PCEYGS3PU3XGLBSN4FLFQTYIEC5HDTC3OQ, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.039163404 (117h46m32.910246449s) [noise=true , address=77.90.31.17:4771 , piece_id=X57ZVYWUTGUUKAOIRL2FL5CHXAN5SQYLX6EJBQIKL4FZGHWGPRQA, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.039588294 (117h46m32.910671339s) [noise=true , address=145.79.227.39:4014 , piece_id=HO5AOQJRC774MOI5S6J7MOHIYV4L72XC22HLQCHBLYP3QYIA2DTQ, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.040016559 (117h46m32.911099594s) [noise=true , address=145.79.227.43:4159 , piece_id=5KI6NRQSBBPKKUN65NU5LTY32YJ65K2OKZDJQEVQX465WNQKNHHQ, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.040450956 (117h46m32.911534001s) [noise=true , address=77.90.31.10:4518 , piece_id=XCU5D2A36LBTDDYTILXWUMMANHVDCW2KMEAB3HUCDU5UONZLULBQ, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.040890654 (117h46m32.911973689s) [noise=true , address=145.79.227.22:4510 , piece_id=ALCQVB6CUD7JAJ7I2NHADFKGHOSFVBKCGR2KHZWHPMYKK3VV4T5A, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.041348756 (117h46m32.912431801s) [noise=true , address=45.91.116.11:4119 , piece_id=S5KLL473HCDKQBM5NVTVNLLSYGMXFFTFDFQX7D5SITM2MQO43K6Q, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.041761201 (117h46m32.912844236s) [noise=true , address=145.79.227.41:4087 , piece_id=OQCN6TONOCOR2EH7THCYREB7LRVYOMK55VPTAWK4NMADKYYYPYLA, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.042078305 (117h46m32.91316134s) [noise=true , address=145.79.227.7:4295 , piece_id=IPMI23BRBMF4MTC6EJJEZWGW7AMFPLGUGMXEFCF2QZASWEUFIH5Q, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.042443110 (117h46m32.913526155s) [noise=true , address=145.79.227.32:4835 , piece_id=4X2XQSZAB6KZPDY5G4VFDFZCL744RAL2IYUYMU7UWFMEII5D225A, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.042561475 (117h46m32.91364452s) [noise=true , address=145.79.227.31:4797 , piece_id=WOGJMBDWY42R5T5T5M2NR4AF3NFZ6CAYOI2O5RRL3JYPCAB3FFZA, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:35.033962808 (117h46m34.905045853s) [upload=102799]: Next returned nil segment Dec 10 13:15:35.033993557 (117h46m34.905076612s) [upload=102799]: Waiting for error group managing segments... Dec 10 13:15:35.355027663 (117h46m35.226110708s) [noise=true , address=77.90.31.10:4518 , piece_id=XCU5D2A36LBTDDYTILXWUMMANHVDCW2KMEAB3HUCDU5UONZLULBQ, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:35.395849857 (117h46m35.266932902s) [noise=true , address=77.90.31.17:4771 , piece_id=X57ZVYWUTGUUKAOIRL2FL5CHXAN5SQYLX6EJBQIKL4FZGHWGPRQA, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:35.939806675 (117h46m35.810889721s) [noise=true , address=145.79.227.31:4797 , piece_id=WOGJMBDWY42R5T5T5M2NR4AF3NFZ6CAYOI2O5RRL3JYPCAB3FFZA, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:35.944824321 (117h46m35.815907356s) [noise=true , address=145.79.227.7:4295 , piece_id=IPMI23BRBMF4MTC6EJJEZWGW7AMFPLGUGMXEFCF2QZASWEUFIH5Q, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:35.966925235 (117h46m35.83800827s) [noise=true , address=145.79.227.41:4087 , piece_id=OQCN6TONOCOR2EH7THCYREB7LRVYOMK55VPTAWK4NMADKYYYPYLA, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:36.183487725 (117h46m36.05457076s) [noise=true , address=145.79.227.29:4943 , piece_id=I5NHKVGJKAG37VAV53QOQ2ZYSDF6SGEXTTG7DEAAG5CC2DO7SVVQ, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:36.184533436 (117h46m36.055616471s) [noise=true , address=145.79.227.22:4509 , piece_id=62EZ3JKY7O5VZEGSATQYYBLKR46ASNJUX3H7YIYAIXAV4WYF26JQ, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:36.188784180 (117h46m36.059867215s) [noise=true , address=145.79.227.6:4259 , piece_id=RQCMHQGVQGAWFBJ6AAHMVON7SLBV7XPYR4D6XRI6DJEJVJ7AFGRQ, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:36.190871565 (117h46m36.0619546s) [noise=true , address=145.79.227.22:4510 , piece_id=ALCQVB6CUD7JAJ7I2NHADFKGHOSFVBKCGR2KHZWHPMYKK3VV4T5A, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:36.234924680 (117h46m36.106007725s) [noise=true , address=145.79.227.39:4014 , piece_id=HO5AOQJRC774MOI5S6J7MOHIYV4L72XC22HLQCHBLYP3QYIA2DTQ, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:37.136102891 (117h46m37.007185926s) [noise=true , address=145.79.227.32:4835 , piece_id=4X2XQSZAB6KZPDY5G4VFDFZCL744RAL2IYUYMU7UWFMEII5D225A, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:37.155928563 (117h46m37.027011618s) [noise=true , address=145.79.227.43:4159 , piece_id=5KI6NRQSBBPKKUN65NU5LTY32YJ65K2OKZDJQEVQX465WNQKNHHQ, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:37.199763735 (117h46m37.0708468s) [noise=true , address=145.79.227.10:4401 , piece_id=S4DH6HYJ5IUWJ5BTURVW7OTDWFS3O3UNCZMSTUK4QZ4OCNKTH6MA, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:37.199789233 (117h46m37.070872279s) [seg_pos={0 0}, upload=102799]: Segment reached optimal threshold of 13 pieces. Dec 10 13:15:37.200041654 (117h46m37.071124689s) [noise=true , address=145.79.227.26:4654 , piece_id=TRV36WT3M3K5LIJB3LCMLG6OFPQWWOMYHFOHGKLRECB7NE5VBEGA, seg_pos={0 0}, upload=102799]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KhBWMerZe9tUmzdVEYkBb6NmJ25dGP26Ea7CLjNJT5gdAeiah): protocol: write tcp 188.166.225.170:53216->145.79.227.26:4654: use of closed network connection; write tcp 188.166.225.170:53216->145.79.227.26:4654: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53216->145.79.227.26:4654: use of closed network connection Dec 10 13:15:37.200055591 (117h46m37.071138656s) [noise=true , address=145.79.227.26:4654 , piece_id=TRV36WT3M3K5LIJB3LCMLG6OFPQWWOMYHFOHGKLRECB7NE5VBEGA, seg_pos={0 0}, upload=102799]: Long tail cancellation due to optimal threshold Dec 10 13:15:37.200156243 (117h46m37.071239288s) [noise=true , address=145.79.227.40:4049 , piece_id=FVAQGZN6T3VF2XVZB4PCEYGS3PU3XGLBSN4FLFQTYIEC5HDTC3OQ, seg_pos={0 0}, upload=102799]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J): protocol: write tcp 188.166.225.170:55280->145.79.227.40:4049: use of closed network connection; write tcp 188.166.225.170:55280->145.79.227.40:4049: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55280->145.79.227.40:4049: use of closed network connection Dec 10 13:15:37.200184998 (117h46m37.071268053s) [noise=true , address=145.79.227.40:4049 , piece_id=FVAQGZN6T3VF2XVZB4PCEYGS3PU3XGLBSN4FLFQTYIEC5HDTC3OQ, seg_pos={0 0}, upload=102799]: Long tail cancellation due to optimal threshold Dec 10 13:15:37.200373426 (117h46m37.071456461s) [noise=true , address=45.91.116.11:4119 , piece_id=S5KLL473HCDKQBM5NVTVNLLSYGMXFFTFDFQX7D5SITM2MQO43K6Q, seg_pos={0 0}, upload=102799]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: expected piece hash; context canceled; EOF Dec 10 13:15:37.200381421 (117h46m37.071464466s) [noise=true , address=45.91.116.11:4119 , piece_id=S5KLL473HCDKQBM5NVTVNLLSYGMXFFTFDFQX7D5SITM2MQO43K6Q, seg_pos={0 0}, upload=102799]: Long tail cancellation due to optimal threshold Dec 10 13:15:37.200402562 (117h46m37.071485597s) [seg_pos={0 0}, upload=102799]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 10 13:15:37.200491951 (117h46m37.071574996s) [upload=102799]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 10 13:15:37.200500108 (117h46m37.071583153s) [upload=102799]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 10 13:15:37.624562388 (117h46m37.495645534s) [upload=102799]: Done uploading segments... Dec 10 13:15:37.624584852 (117h46m37.495667887s) [upload=102799]: Upload finished. err: <nil> Duration: 27.677382057s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 55 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXX..X.XXXX.XXXXXXXXXXXX.XXXXXXXXXXXXXXXXX] duration: 5.001295183s trace: https://jaeger.dev.storj.io/trace/7b6ea66bceb73d5f FAIL (2025-12-10 13:31:30 UTC) orbiter warning: context deadline exceeded Duration: 5.001320121s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [...X...........................................X..X.X....X..] duration: 6.100064952s trace: https://jaeger.dev.storj.io/trace/70ffd77b4c08e04c FAIL (2025-12-10 13:31:06 UTC) orbiter warning: context deadline exceeded Duration: 6.100093785s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 55 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXX..X.XXXX.XXXXXXXXXXXX.XXXXXXXXXXXXXXXX] duration: 5.001531845s trace: https://jaeger.dev.storj.io/trace/1042768c52ad4300 FAIL (2025-12-10 13:30:57 UTC) orbiter warning: context deadline exceeded Duration: 5.001556912s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [....X...........................................X..X.X....X.] duration: 6.408318588s trace: https://jaeger.dev.storj.io/trace/70ffd77b4c08e04c FAIL (2025-12-10 13:30:31 UTC) orbiter warning: context deadline exceeded Duration: 6.408346532s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.X...X.................................X....................] duration: 22.913956923s trace: https://jaeger.dev.storj.io/trace/19144dc231ae8183 FAIL (2025-12-10 13:30:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ts... Dec 10 13:15:32.674883550 (117h46m32.545966595s) [upload=102799]: Got next segment. Inline: false Dec 10 13:15:32.674940880 (117h46m32.546023915s) [upload=102799]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 10 13:15:32.674946670 (117h46m32.546029706s) [upload=102799]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 10 13:15:33.036416584 (117h46m32.907499629s) [seg_pos={0 0}, upload=102799]: Begin upload segment... Dec 10 13:15:33.036472831 (117h46m32.907555876s) [seg_pos={0 0}, upload=102799]: Done begin upload segment. Dec 10 13:15:33.036565578 (117h46m32.907648613s) [noise=true , address=145.79.227.29:4943 , piece_id=I5NHKVGJKAG37VAV53QOQ2ZYSDF6SGEXTTG7DEAAG5CC2DO7SVVQ, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.036606606 (117h46m32.907689651s) [noise=true , address=145.79.227.6:4259 , piece_id=RQCMHQGVQGAWFBJ6AAHMVON7SLBV7XPYR4D6XRI6DJEJVJ7AFGRQ, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.037385819 (117h46m32.908468865s) [noise=true , address=145.79.227.26:4654 , piece_id=TRV36WT3M3K5LIJB3LCMLG6OFPQWWOMYHFOHGKLRECB7NE5VBEGA, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.037833743 (117h46m32.908916778s) [noise=true , address=145.79.227.22:4509 , piece_id=62EZ3JKY7O5VZEGSATQYYBLKR46ASNJUX3H7YIYAIXAV4WYF26JQ, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.038339726 (117h46m32.909422772s) [noise=true , address=145.79.227.10:4401 , piece_id=S4DH6HYJ5IUWJ5BTURVW7OTDWFS3O3UNCZMSTUK4QZ4OCNKTH6MA, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.038758464 (117h46m32.909841509s) [noise=true , address=145.79.227.40:4049 , piece_id=FVAQGZN6T3VF2XVZB4PCEYGS3PU3XGLBSN4FLFQTYIEC5HDTC3OQ, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.039163404 (117h46m32.910246449s) [noise=true , address=77.90.31.17:4771 , piece_id=X57ZVYWUTGUUKAOIRL2FL5CHXAN5SQYLX6EJBQIKL4FZGHWGPRQA, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.039588294 (117h46m32.910671339s) [noise=true , address=145.79.227.39:4014 , piece_id=HO5AOQJRC774MOI5S6J7MOHIYV4L72XC22HLQCHBLYP3QYIA2DTQ, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.040016559 (117h46m32.911099594s) [noise=true , address=145.79.227.43:4159 , piece_id=5KI6NRQSBBPKKUN65NU5LTY32YJ65K2OKZDJQEVQX465WNQKNHHQ, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.040450956 (117h46m32.911534001s) [noise=true , address=77.90.31.10:4518 , piece_id=XCU5D2A36LBTDDYTILXWUMMANHVDCW2KMEAB3HUCDU5UONZLULBQ, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.040890654 (117h46m32.911973689s) [noise=true , address=145.79.227.22:4510 , piece_id=ALCQVB6CUD7JAJ7I2NHADFKGHOSFVBKCGR2KHZWHPMYKK3VV4T5A, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.041348756 (117h46m32.912431801s) [noise=true , address=45.91.116.11:4119 , piece_id=S5KLL473HCDKQBM5NVTVNLLSYGMXFFTFDFQX7D5SITM2MQO43K6Q, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.041761201 (117h46m32.912844236s) [noise=true , address=145.79.227.41:4087 , piece_id=OQCN6TONOCOR2EH7THCYREB7LRVYOMK55VPTAWK4NMADKYYYPYLA, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.042078305 (117h46m32.91316134s) [noise=true , address=145.79.227.7:4295 , piece_id=IPMI23BRBMF4MTC6EJJEZWGW7AMFPLGUGMXEFCF2QZASWEUFIH5Q, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.042443110 (117h46m32.913526155s) [noise=true , address=145.79.227.32:4835 , piece_id=4X2XQSZAB6KZPDY5G4VFDFZCL744RAL2IYUYMU7UWFMEII5D225A, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.042561475 (117h46m32.91364452s) [noise=true , address=145.79.227.31:4797 , piece_id=WOGJMBDWY42R5T5T5M2NR4AF3NFZ6CAYOI2O5RRL3JYPCAB3FFZA, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:35.033962808 (117h46m34.905045853s) [upload=102799]: Next returned nil segment Dec 10 13:15:35.033993557 (117h46m34.905076612s) [upload=102799]: Waiting for error group managing segments... Dec 10 13:15:35.355027663 (117h46m35.226110708s) [noise=true , address=77.90.31.10:4518 , piece_id=XCU5D2A36LBTDDYTILXWUMMANHVDCW2KMEAB3HUCDU5UONZLULBQ, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:35.395849857 (117h46m35.266932902s) [noise=true , address=77.90.31.17:4771 , piece_id=X57ZVYWUTGUUKAOIRL2FL5CHXAN5SQYLX6EJBQIKL4FZGHWGPRQA, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:35.939806675 (117h46m35.810889721s) [noise=true , address=145.79.227.31:4797 , piece_id=WOGJMBDWY42R5T5T5M2NR4AF3NFZ6CAYOI2O5RRL3JYPCAB3FFZA, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:35.944824321 (117h46m35.815907356s) [noise=true , address=145.79.227.7:4295 , piece_id=IPMI23BRBMF4MTC6EJJEZWGW7AMFPLGUGMXEFCF2QZASWEUFIH5Q, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:35.966925235 (117h46m35.83800827s) [noise=true , address=145.79.227.41:4087 , piece_id=OQCN6TONOCOR2EH7THCYREB7LRVYOMK55VPTAWK4NMADKYYYPYLA, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:36.183487725 (117h46m36.05457076s) [noise=true , address=145.79.227.29:4943 , piece_id=I5NHKVGJKAG37VAV53QOQ2ZYSDF6SGEXTTG7DEAAG5CC2DO7SVVQ, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:36.184533436 (117h46m36.055616471s) [noise=true , address=145.79.227.22:4509 , piece_id=62EZ3JKY7O5VZEGSATQYYBLKR46ASNJUX3H7YIYAIXAV4WYF26JQ, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:36.188784180 (117h46m36.059867215s) [noise=true , address=145.79.227.6:4259 , piece_id=RQCMHQGVQGAWFBJ6AAHMVON7SLBV7XPYR4D6XRI6DJEJVJ7AFGRQ, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:36.190871565 (117h46m36.0619546s) [noise=true , address=145.79.227.22:4510 , piece_id=ALCQVB6CUD7JAJ7I2NHADFKGHOSFVBKCGR2KHZWHPMYKK3VV4T5A, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:36.234924680 (117h46m36.106007725s) [noise=true , address=145.79.227.39:4014 , piece_id=HO5AOQJRC774MOI5S6J7MOHIYV4L72XC22HLQCHBLYP3QYIA2DTQ, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:37.136102891 (117h46m37.007185926s) [noise=true , address=145.79.227.32:4835 , piece_id=4X2XQSZAB6KZPDY5G4VFDFZCL744RAL2IYUYMU7UWFMEII5D225A, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:37.155928563 (117h46m37.027011618s) [noise=true , address=145.79.227.43:4159 , piece_id=5KI6NRQSBBPKKUN65NU5LTY32YJ65K2OKZDJQEVQX465WNQKNHHQ, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:37.199763735 (117h46m37.0708468s) [noise=true , address=145.79.227.10:4401 , piece_id=S4DH6HYJ5IUWJ5BTURVW7OTDWFS3O3UNCZMSTUK4QZ4OCNKTH6MA, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:37.199789233 (117h46m37.070872279s) [seg_pos={0 0}, upload=102799]: Segment reached optimal threshold of 13 pieces. Dec 10 13:15:37.200041654 (117h46m37.071124689s) [noise=true , address=145.79.227.26:4654 , piece_id=TRV36WT3M3K5LIJB3LCMLG6OFPQWWOMYHFOHGKLRECB7NE5VBEGA, seg_pos={0 0}, upload=102799]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KhBWMerZe9tUmzdVEYkBb6NmJ25dGP26Ea7CLjNJT5gdAeiah): protocol: write tcp 188.166.225.170:53216->145.79.227.26:4654: use of closed network connection; write tcp 188.166.225.170:53216->145.79.227.26:4654: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53216->145.79.227.26:4654: use of closed network connection Dec 10 13:15:37.200055591 (117h46m37.071138656s) [noise=true , address=145.79.227.26:4654 , piece_id=TRV36WT3M3K5LIJB3LCMLG6OFPQWWOMYHFOHGKLRECB7NE5VBEGA, seg_pos={0 0}, upload=102799]: Long tail cancellation due to optimal threshold Dec 10 13:15:37.200156243 (117h46m37.071239288s) [noise=true , address=145.79.227.40:4049 , piece_id=FVAQGZN6T3VF2XVZB4PCEYGS3PU3XGLBSN4FLFQTYIEC5HDTC3OQ, seg_pos={0 0}, upload=102799]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J): protocol: write tcp 188.166.225.170:55280->145.79.227.40:4049: use of closed network connection; write tcp 188.166.225.170:55280->145.79.227.40:4049: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55280->145.79.227.40:4049: use of closed network connection Dec 10 13:15:37.200184998 (117h46m37.071268053s) [noise=true , address=145.79.227.40:4049 , piece_id=FVAQGZN6T3VF2XVZB4PCEYGS3PU3XGLBSN4FLFQTYIEC5HDTC3OQ, seg_pos={0 0}, upload=102799]: Long tail cancellation due to optimal threshold Dec 10 13:15:37.200373426 (117h46m37.071456461s) [noise=true , address=45.91.116.11:4119 , piece_id=S5KLL473HCDKQBM5NVTVNLLSYGMXFFTFDFQX7D5SITM2MQO43K6Q, seg_pos={0 0}, upload=102799]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: expected piece hash; context canceled; EOF Dec 10 13:15:37.200381421 (117h46m37.071464466s) [noise=true , address=45.91.116.11:4119 , piece_id=S5KLL473HCDKQBM5NVTVNLLSYGMXFFTFDFQX7D5SITM2MQO43K6Q, seg_pos={0 0}, upload=102799]: Long tail cancellation due to optimal threshold Dec 10 13:15:37.200402562 (117h46m37.071485597s) [seg_pos={0 0}, upload=102799]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 10 13:15:37.200491951 (117h46m37.071574996s) [upload=102799]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 10 13:15:37.200500108 (117h46m37.071583153s) [upload=102799]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 10 13:15:37.624562388 (117h46m37.495645534s) [upload=102799]: Done uploading segments... Dec 10 13:15:37.624584852 (117h46m37.495667887s) [upload=102799]: Upload finished. err: <nil> Duration: 22.91399158s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 4.556106564s trace: https://jaeger.dev.storj.io/trace/a0b5a94505b001 FAIL (2025-12-10 13:30:25 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:36068->45.38.24.2:443: read: connection reset by peer Duration: 4.55619865s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.....X...........................................X..X.X....X] duration: 30.001024201s trace: https://jaeger.dev.storj.io/trace/70ffd77b4c08e04c FAIL (2025-12-10 13:30:14 UTC) orbiter warning: context deadline exceeded Duration: 30.001051693s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 55 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXX..X.XXXX.XXXXXXXXXXXX.XXXXXXXXXXXXXXX] duration: 5.001420142s trace: https://jaeger.dev.storj.io/trace/24c0307d97c3c60b FAIL (2025-12-10 13:30:06 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001446883s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 55 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXXX..X.XXXX.XXXXXXXXXXXX.XXXXXXXXXXXXXX] duration: 5.000968921s trace: https://jaeger.dev.storj.io/trace/6bb2b40440a6e045 FAIL (2025-12-10 13:29:31 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000995671s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..X...X.................................X...................] duration: 10.133813001s trace: https://jaeger.dev.storj.io/trace/19144dc231ae8183 FAIL (2025-12-10 13:29:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ts... Dec 10 13:15:32.674883550 (117h46m32.545966595s) [upload=102799]: Got next segment. Inline: false Dec 10 13:15:32.674940880 (117h46m32.546023915s) [upload=102799]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 10 13:15:32.674946670 (117h46m32.546029706s) [upload=102799]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 10 13:15:33.036416584 (117h46m32.907499629s) [seg_pos={0 0}, upload=102799]: Begin upload segment... Dec 10 13:15:33.036472831 (117h46m32.907555876s) [seg_pos={0 0}, upload=102799]: Done begin upload segment. Dec 10 13:15:33.036565578 (117h46m32.907648613s) [noise=true , address=145.79.227.29:4943 , piece_id=I5NHKVGJKAG37VAV53QOQ2ZYSDF6SGEXTTG7DEAAG5CC2DO7SVVQ, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.036606606 (117h46m32.907689651s) [noise=true , address=145.79.227.6:4259 , piece_id=RQCMHQGVQGAWFBJ6AAHMVON7SLBV7XPYR4D6XRI6DJEJVJ7AFGRQ, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.037385819 (117h46m32.908468865s) [noise=true , address=145.79.227.26:4654 , piece_id=TRV36WT3M3K5LIJB3LCMLG6OFPQWWOMYHFOHGKLRECB7NE5VBEGA, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.037833743 (117h46m32.908916778s) [noise=true , address=145.79.227.22:4509 , piece_id=62EZ3JKY7O5VZEGSATQYYBLKR46ASNJUX3H7YIYAIXAV4WYF26JQ, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.038339726 (117h46m32.909422772s) [noise=true , address=145.79.227.10:4401 , piece_id=S4DH6HYJ5IUWJ5BTURVW7OTDWFS3O3UNCZMSTUK4QZ4OCNKTH6MA, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.038758464 (117h46m32.909841509s) [noise=true , address=145.79.227.40:4049 , piece_id=FVAQGZN6T3VF2XVZB4PCEYGS3PU3XGLBSN4FLFQTYIEC5HDTC3OQ, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.039163404 (117h46m32.910246449s) [noise=true , address=77.90.31.17:4771 , piece_id=X57ZVYWUTGUUKAOIRL2FL5CHXAN5SQYLX6EJBQIKL4FZGHWGPRQA, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.039588294 (117h46m32.910671339s) [noise=true , address=145.79.227.39:4014 , piece_id=HO5AOQJRC774MOI5S6J7MOHIYV4L72XC22HLQCHBLYP3QYIA2DTQ, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.040016559 (117h46m32.911099594s) [noise=true , address=145.79.227.43:4159 , piece_id=5KI6NRQSBBPKKUN65NU5LTY32YJ65K2OKZDJQEVQX465WNQKNHHQ, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.040450956 (117h46m32.911534001s) [noise=true , address=77.90.31.10:4518 , piece_id=XCU5D2A36LBTDDYTILXWUMMANHVDCW2KMEAB3HUCDU5UONZLULBQ, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.040890654 (117h46m32.911973689s) [noise=true , address=145.79.227.22:4510 , piece_id=ALCQVB6CUD7JAJ7I2NHADFKGHOSFVBKCGR2KHZWHPMYKK3VV4T5A, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.041348756 (117h46m32.912431801s) [noise=true , address=45.91.116.11:4119 , piece_id=S5KLL473HCDKQBM5NVTVNLLSYGMXFFTFDFQX7D5SITM2MQO43K6Q, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.041761201 (117h46m32.912844236s) [noise=true , address=145.79.227.41:4087 , piece_id=OQCN6TONOCOR2EH7THCYREB7LRVYOMK55VPTAWK4NMADKYYYPYLA, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.042078305 (117h46m32.91316134s) [noise=true , address=145.79.227.7:4295 , piece_id=IPMI23BRBMF4MTC6EJJEZWGW7AMFPLGUGMXEFCF2QZASWEUFIH5Q, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.042443110 (117h46m32.913526155s) [noise=true , address=145.79.227.32:4835 , piece_id=4X2XQSZAB6KZPDY5G4VFDFZCL744RAL2IYUYMU7UWFMEII5D225A, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.042561475 (117h46m32.91364452s) [noise=true , address=145.79.227.31:4797 , piece_id=WOGJMBDWY42R5T5T5M2NR4AF3NFZ6CAYOI2O5RRL3JYPCAB3FFZA, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:35.033962808 (117h46m34.905045853s) [upload=102799]: Next returned nil segment Dec 10 13:15:35.033993557 (117h46m34.905076612s) [upload=102799]: Waiting for error group managing segments... Dec 10 13:15:35.355027663 (117h46m35.226110708s) [noise=true , address=77.90.31.10:4518 , piece_id=XCU5D2A36LBTDDYTILXWUMMANHVDCW2KMEAB3HUCDU5UONZLULBQ, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:35.395849857 (117h46m35.266932902s) [noise=true , address=77.90.31.17:4771 , piece_id=X57ZVYWUTGUUKAOIRL2FL5CHXAN5SQYLX6EJBQIKL4FZGHWGPRQA, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:35.939806675 (117h46m35.810889721s) [noise=true , address=145.79.227.31:4797 , piece_id=WOGJMBDWY42R5T5T5M2NR4AF3NFZ6CAYOI2O5RRL3JYPCAB3FFZA, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:35.944824321 (117h46m35.815907356s) [noise=true , address=145.79.227.7:4295 , piece_id=IPMI23BRBMF4MTC6EJJEZWGW7AMFPLGUGMXEFCF2QZASWEUFIH5Q, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:35.966925235 (117h46m35.83800827s) [noise=true , address=145.79.227.41:4087 , piece_id=OQCN6TONOCOR2EH7THCYREB7LRVYOMK55VPTAWK4NMADKYYYPYLA, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:36.183487725 (117h46m36.05457076s) [noise=true , address=145.79.227.29:4943 , piece_id=I5NHKVGJKAG37VAV53QOQ2ZYSDF6SGEXTTG7DEAAG5CC2DO7SVVQ, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:36.184533436 (117h46m36.055616471s) [noise=true , address=145.79.227.22:4509 , piece_id=62EZ3JKY7O5VZEGSATQYYBLKR46ASNJUX3H7YIYAIXAV4WYF26JQ, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:36.188784180 (117h46m36.059867215s) [noise=true , address=145.79.227.6:4259 , piece_id=RQCMHQGVQGAWFBJ6AAHMVON7SLBV7XPYR4D6XRI6DJEJVJ7AFGRQ, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:36.190871565 (117h46m36.0619546s) [noise=true , address=145.79.227.22:4510 , piece_id=ALCQVB6CUD7JAJ7I2NHADFKGHOSFVBKCGR2KHZWHPMYKK3VV4T5A, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:36.234924680 (117h46m36.106007725s) [noise=true , address=145.79.227.39:4014 , piece_id=HO5AOQJRC774MOI5S6J7MOHIYV4L72XC22HLQCHBLYP3QYIA2DTQ, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:37.136102891 (117h46m37.007185926s) [noise=true , address=145.79.227.32:4835 , piece_id=4X2XQSZAB6KZPDY5G4VFDFZCL744RAL2IYUYMU7UWFMEII5D225A, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:37.155928563 (117h46m37.027011618s) [noise=true , address=145.79.227.43:4159 , piece_id=5KI6NRQSBBPKKUN65NU5LTY32YJ65K2OKZDJQEVQX465WNQKNHHQ, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:37.199763735 (117h46m37.0708468s) [noise=true , address=145.79.227.10:4401 , piece_id=S4DH6HYJ5IUWJ5BTURVW7OTDWFS3O3UNCZMSTUK4QZ4OCNKTH6MA, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:37.199789233 (117h46m37.070872279s) [seg_pos={0 0}, upload=102799]: Segment reached optimal threshold of 13 pieces. Dec 10 13:15:37.200041654 (117h46m37.071124689s) [noise=true , address=145.79.227.26:4654 , piece_id=TRV36WT3M3K5LIJB3LCMLG6OFPQWWOMYHFOHGKLRECB7NE5VBEGA, seg_pos={0 0}, upload=102799]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KhBWMerZe9tUmzdVEYkBb6NmJ25dGP26Ea7CLjNJT5gdAeiah): protocol: write tcp 188.166.225.170:53216->145.79.227.26:4654: use of closed network connection; write tcp 188.166.225.170:53216->145.79.227.26:4654: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53216->145.79.227.26:4654: use of closed network connection Dec 10 13:15:37.200055591 (117h46m37.071138656s) [noise=true , address=145.79.227.26:4654 , piece_id=TRV36WT3M3K5LIJB3LCMLG6OFPQWWOMYHFOHGKLRECB7NE5VBEGA, seg_pos={0 0}, upload=102799]: Long tail cancellation due to optimal threshold Dec 10 13:15:37.200156243 (117h46m37.071239288s) [noise=true , address=145.79.227.40:4049 , piece_id=FVAQGZN6T3VF2XVZB4PCEYGS3PU3XGLBSN4FLFQTYIEC5HDTC3OQ, seg_pos={0 0}, upload=102799]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J): protocol: write tcp 188.166.225.170:55280->145.79.227.40:4049: use of closed network connection; write tcp 188.166.225.170:55280->145.79.227.40:4049: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55280->145.79.227.40:4049: use of closed network connection Dec 10 13:15:37.200184998 (117h46m37.071268053s) [noise=true , address=145.79.227.40:4049 , piece_id=FVAQGZN6T3VF2XVZB4PCEYGS3PU3XGLBSN4FLFQTYIEC5HDTC3OQ, seg_pos={0 0}, upload=102799]: Long tail cancellation due to optimal threshold Dec 10 13:15:37.200373426 (117h46m37.071456461s) [noise=true , address=45.91.116.11:4119 , piece_id=S5KLL473HCDKQBM5NVTVNLLSYGMXFFTFDFQX7D5SITM2MQO43K6Q, seg_pos={0 0}, upload=102799]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: expected piece hash; context canceled; EOF Dec 10 13:15:37.200381421 (117h46m37.071464466s) [noise=true , address=45.91.116.11:4119 , piece_id=S5KLL473HCDKQBM5NVTVNLLSYGMXFFTFDFQX7D5SITM2MQO43K6Q, seg_pos={0 0}, upload=102799]: Long tail cancellation due to optimal threshold Dec 10 13:15:37.200402562 (117h46m37.071485597s) [seg_pos={0 0}, upload=102799]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 10 13:15:37.200491951 (117h46m37.071574996s) [upload=102799]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 10 13:15:37.200500108 (117h46m37.071583153s) [upload=102799]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 10 13:15:37.624562388 (117h46m37.495645534s) [upload=102799]: Done uploading segments... Dec 10 13:15:37.624584852 (117h46m37.495667887s) [upload=102799]: Upload finished. err: <nil> Duration: 10.133904006s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 3.506820915s trace: https://jaeger.dev.storj.io/trace/a0b5a94505b001 FAIL (2025-12-10 13:29:20 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:36068->45.38.24.2:443: read: connection reset by peer Duration: 3.506846533s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...X...X.................................X..................] duration: 8.798523786s trace: https://jaeger.dev.storj.io/trace/19144dc231ae8183 FAIL (2025-12-10 13:28:49 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ts... Dec 10 13:15:32.674883550 (117h46m32.545966595s) [upload=102799]: Got next segment. Inline: false Dec 10 13:15:32.674940880 (117h46m32.546023915s) [upload=102799]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 10 13:15:32.674946670 (117h46m32.546029706s) [upload=102799]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 10 13:15:33.036416584 (117h46m32.907499629s) [seg_pos={0 0}, upload=102799]: Begin upload segment... Dec 10 13:15:33.036472831 (117h46m32.907555876s) [seg_pos={0 0}, upload=102799]: Done begin upload segment. Dec 10 13:15:33.036565578 (117h46m32.907648613s) [noise=true , address=145.79.227.29:4943 , piece_id=I5NHKVGJKAG37VAV53QOQ2ZYSDF6SGEXTTG7DEAAG5CC2DO7SVVQ, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.036606606 (117h46m32.907689651s) [noise=true , address=145.79.227.6:4259 , piece_id=RQCMHQGVQGAWFBJ6AAHMVON7SLBV7XPYR4D6XRI6DJEJVJ7AFGRQ, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.037385819 (117h46m32.908468865s) [noise=true , address=145.79.227.26:4654 , piece_id=TRV36WT3M3K5LIJB3LCMLG6OFPQWWOMYHFOHGKLRECB7NE5VBEGA, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.037833743 (117h46m32.908916778s) [noise=true , address=145.79.227.22:4509 , piece_id=62EZ3JKY7O5VZEGSATQYYBLKR46ASNJUX3H7YIYAIXAV4WYF26JQ, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.038339726 (117h46m32.909422772s) [noise=true , address=145.79.227.10:4401 , piece_id=S4DH6HYJ5IUWJ5BTURVW7OTDWFS3O3UNCZMSTUK4QZ4OCNKTH6MA, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.038758464 (117h46m32.909841509s) [noise=true , address=145.79.227.40:4049 , piece_id=FVAQGZN6T3VF2XVZB4PCEYGS3PU3XGLBSN4FLFQTYIEC5HDTC3OQ, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.039163404 (117h46m32.910246449s) [noise=true , address=77.90.31.17:4771 , piece_id=X57ZVYWUTGUUKAOIRL2FL5CHXAN5SQYLX6EJBQIKL4FZGHWGPRQA, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.039588294 (117h46m32.910671339s) [noise=true , address=145.79.227.39:4014 , piece_id=HO5AOQJRC774MOI5S6J7MOHIYV4L72XC22HLQCHBLYP3QYIA2DTQ, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.040016559 (117h46m32.911099594s) [noise=true , address=145.79.227.43:4159 , piece_id=5KI6NRQSBBPKKUN65NU5LTY32YJ65K2OKZDJQEVQX465WNQKNHHQ, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.040450956 (117h46m32.911534001s) [noise=true , address=77.90.31.10:4518 , piece_id=XCU5D2A36LBTDDYTILXWUMMANHVDCW2KMEAB3HUCDU5UONZLULBQ, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.040890654 (117h46m32.911973689s) [noise=true , address=145.79.227.22:4510 , piece_id=ALCQVB6CUD7JAJ7I2NHADFKGHOSFVBKCGR2KHZWHPMYKK3VV4T5A, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.041348756 (117h46m32.912431801s) [noise=true , address=45.91.116.11:4119 , piece_id=S5KLL473HCDKQBM5NVTVNLLSYGMXFFTFDFQX7D5SITM2MQO43K6Q, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.041761201 (117h46m32.912844236s) [noise=true , address=145.79.227.41:4087 , piece_id=OQCN6TONOCOR2EH7THCYREB7LRVYOMK55VPTAWK4NMADKYYYPYLA, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.042078305 (117h46m32.91316134s) [noise=true , address=145.79.227.7:4295 , piece_id=IPMI23BRBMF4MTC6EJJEZWGW7AMFPLGUGMXEFCF2QZASWEUFIH5Q, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.042443110 (117h46m32.913526155s) [noise=true , address=145.79.227.32:4835 , piece_id=4X2XQSZAB6KZPDY5G4VFDFZCL744RAL2IYUYMU7UWFMEII5D225A, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.042561475 (117h46m32.91364452s) [noise=true , address=145.79.227.31:4797 , piece_id=WOGJMBDWY42R5T5T5M2NR4AF3NFZ6CAYOI2O5RRL3JYPCAB3FFZA, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:35.033962808 (117h46m34.905045853s) [upload=102799]: Next returned nil segment Dec 10 13:15:35.033993557 (117h46m34.905076612s) [upload=102799]: Waiting for error group managing segments... Dec 10 13:15:35.355027663 (117h46m35.226110708s) [noise=true , address=77.90.31.10:4518 , piece_id=XCU5D2A36LBTDDYTILXWUMMANHVDCW2KMEAB3HUCDU5UONZLULBQ, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:35.395849857 (117h46m35.266932902s) [noise=true , address=77.90.31.17:4771 , piece_id=X57ZVYWUTGUUKAOIRL2FL5CHXAN5SQYLX6EJBQIKL4FZGHWGPRQA, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:35.939806675 (117h46m35.810889721s) [noise=true , address=145.79.227.31:4797 , piece_id=WOGJMBDWY42R5T5T5M2NR4AF3NFZ6CAYOI2O5RRL3JYPCAB3FFZA, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:35.944824321 (117h46m35.815907356s) [noise=true , address=145.79.227.7:4295 , piece_id=IPMI23BRBMF4MTC6EJJEZWGW7AMFPLGUGMXEFCF2QZASWEUFIH5Q, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:35.966925235 (117h46m35.83800827s) [noise=true , address=145.79.227.41:4087 , piece_id=OQCN6TONOCOR2EH7THCYREB7LRVYOMK55VPTAWK4NMADKYYYPYLA, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:36.183487725 (117h46m36.05457076s) [noise=true , address=145.79.227.29:4943 , piece_id=I5NHKVGJKAG37VAV53QOQ2ZYSDF6SGEXTTG7DEAAG5CC2DO7SVVQ, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:36.184533436 (117h46m36.055616471s) [noise=true , address=145.79.227.22:4509 , piece_id=62EZ3JKY7O5VZEGSATQYYBLKR46ASNJUX3H7YIYAIXAV4WYF26JQ, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:36.188784180 (117h46m36.059867215s) [noise=true , address=145.79.227.6:4259 , piece_id=RQCMHQGVQGAWFBJ6AAHMVON7SLBV7XPYR4D6XRI6DJEJVJ7AFGRQ, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:36.190871565 (117h46m36.0619546s) [noise=true , address=145.79.227.22:4510 , piece_id=ALCQVB6CUD7JAJ7I2NHADFKGHOSFVBKCGR2KHZWHPMYKK3VV4T5A, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:36.234924680 (117h46m36.106007725s) [noise=true , address=145.79.227.39:4014 , piece_id=HO5AOQJRC774MOI5S6J7MOHIYV4L72XC22HLQCHBLYP3QYIA2DTQ, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:37.136102891 (117h46m37.007185926s) [noise=true , address=145.79.227.32:4835 , piece_id=4X2XQSZAB6KZPDY5G4VFDFZCL744RAL2IYUYMU7UWFMEII5D225A, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:37.155928563 (117h46m37.027011618s) [noise=true , address=145.79.227.43:4159 , piece_id=5KI6NRQSBBPKKUN65NU5LTY32YJ65K2OKZDJQEVQX465WNQKNHHQ, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:37.199763735 (117h46m37.0708468s) [noise=true , address=145.79.227.10:4401 , piece_id=S4DH6HYJ5IUWJ5BTURVW7OTDWFS3O3UNCZMSTUK4QZ4OCNKTH6MA, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:37.199789233 (117h46m37.070872279s) [seg_pos={0 0}, upload=102799]: Segment reached optimal threshold of 13 pieces. Dec 10 13:15:37.200041654 (117h46m37.071124689s) [noise=true , address=145.79.227.26:4654 , piece_id=TRV36WT3M3K5LIJB3LCMLG6OFPQWWOMYHFOHGKLRECB7NE5VBEGA, seg_pos={0 0}, upload=102799]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KhBWMerZe9tUmzdVEYkBb6NmJ25dGP26Ea7CLjNJT5gdAeiah): protocol: write tcp 188.166.225.170:53216->145.79.227.26:4654: use of closed network connection; write tcp 188.166.225.170:53216->145.79.227.26:4654: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53216->145.79.227.26:4654: use of closed network connection Dec 10 13:15:37.200055591 (117h46m37.071138656s) [noise=true , address=145.79.227.26:4654 , piece_id=TRV36WT3M3K5LIJB3LCMLG6OFPQWWOMYHFOHGKLRECB7NE5VBEGA, seg_pos={0 0}, upload=102799]: Long tail cancellation due to optimal threshold Dec 10 13:15:37.200156243 (117h46m37.071239288s) [noise=true , address=145.79.227.40:4049 , piece_id=FVAQGZN6T3VF2XVZB4PCEYGS3PU3XGLBSN4FLFQTYIEC5HDTC3OQ, seg_pos={0 0}, upload=102799]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J): protocol: write tcp 188.166.225.170:55280->145.79.227.40:4049: use of closed network connection; write tcp 188.166.225.170:55280->145.79.227.40:4049: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55280->145.79.227.40:4049: use of closed network connection Dec 10 13:15:37.200184998 (117h46m37.071268053s) [noise=true , address=145.79.227.40:4049 , piece_id=FVAQGZN6T3VF2XVZB4PCEYGS3PU3XGLBSN4FLFQTYIEC5HDTC3OQ, seg_pos={0 0}, upload=102799]: Long tail cancellation due to optimal threshold Dec 10 13:15:37.200373426 (117h46m37.071456461s) [noise=true , address=45.91.116.11:4119 , piece_id=S5KLL473HCDKQBM5NVTVNLLSYGMXFFTFDFQX7D5SITM2MQO43K6Q, seg_pos={0 0}, upload=102799]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: expected piece hash; context canceled; EOF Dec 10 13:15:37.200381421 (117h46m37.071464466s) [noise=true , address=45.91.116.11:4119 , piece_id=S5KLL473HCDKQBM5NVTVNLLSYGMXFFTFDFQX7D5SITM2MQO43K6Q, seg_pos={0 0}, upload=102799]: Long tail cancellation due to optimal threshold Dec 10 13:15:37.200402562 (117h46m37.071485597s) [seg_pos={0 0}, upload=102799]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 10 13:15:37.200491951 (117h46m37.071574996s) [upload=102799]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 10 13:15:37.200500108 (117h46m37.071583153s) [upload=102799]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 10 13:15:37.624562388 (117h46m37.495645534s) [upload=102799]: Done uploading segments... Dec 10 13:15:37.624584852 (117h46m37.495667887s) [upload=102799]: Upload finished. err: <nil> Duration: 8.798548703s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 54 out of 60 is failed (threshold=6) [.XXXXXXXXXXXXXXXXXXXXXXXX..X.XXXX.XXXXXXXXXXXX.XXXXXXXXXXXXX] duration: 5.001691127s trace: https://jaeger.dev.storj.io/trace/d8643ba07b2f310 FAIL (2025-12-10 13:28:42 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001719142s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 4.08199711s trace: https://jaeger.dev.storj.io/trace/a0b5a94505b001 FAIL (2025-12-10 13:28:40 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:36068->45.38.24.2:443: read: connection reset by peer Duration: 4.082120987s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [......X...........................................X..X.X....] duration: 6.721287156s trace: https://jaeger.dev.storj.io/trace/4e7e66895306662b FAIL (2025-12-10 13:28:36 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.721442683s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 54 out of 60 is failed (threshold=6) [X.XXXXXXXXXXXXXXXXXXXXXXXX..X.XXXX.XXXXXXXXXXXX.XXXXXXXXXXXX] duration: 5.002547128s trace: https://jaeger.dev.storj.io/trace/299b53918ac684f6 FAIL (2025-12-10 13:28:17 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.002574941s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.......X...........................................X..X.X...] duration: 12.241092486s trace: https://jaeger.dev.storj.io/trace/4e7e66895306662b FAIL (2025-12-10 13:28:16 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 12.241123975s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 3.459774699s trace: https://jaeger.dev.storj.io/trace/a0b5a94505b001 FAIL (2025-12-10 13:28:13 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:36068->45.38.24.2:443: read: connection reset by peer Duration: 3.459807442s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 53 out of 60 is failed (threshold=6) [.X.XXXXXXXXXXXXXXXXXXXXXXXX..X.XXXX.XXXXXXXXXXXX.XXXXXXXXXXX] duration: 5.001675792s trace: https://jaeger.dev.storj.io/trace/c403e864f70d14b FAIL (2025-12-10 13:28:01 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001697502s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [....X...X.................................X.................] duration: 14.089055896s trace: https://jaeger.dev.storj.io/trace/19144dc231ae8183 FAIL (2025-12-10 13:27:57 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ts... Dec 10 13:15:32.674883550 (117h46m32.545966595s) [upload=102799]: Got next segment. Inline: false Dec 10 13:15:32.674940880 (117h46m32.546023915s) [upload=102799]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 10 13:15:32.674946670 (117h46m32.546029706s) [upload=102799]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 10 13:15:33.036416584 (117h46m32.907499629s) [seg_pos={0 0}, upload=102799]: Begin upload segment... Dec 10 13:15:33.036472831 (117h46m32.907555876s) [seg_pos={0 0}, upload=102799]: Done begin upload segment. Dec 10 13:15:33.036565578 (117h46m32.907648613s) [noise=true , address=145.79.227.29:4943 , piece_id=I5NHKVGJKAG37VAV53QOQ2ZYSDF6SGEXTTG7DEAAG5CC2DO7SVVQ, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.036606606 (117h46m32.907689651s) [noise=true , address=145.79.227.6:4259 , piece_id=RQCMHQGVQGAWFBJ6AAHMVON7SLBV7XPYR4D6XRI6DJEJVJ7AFGRQ, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.037385819 (117h46m32.908468865s) [noise=true , address=145.79.227.26:4654 , piece_id=TRV36WT3M3K5LIJB3LCMLG6OFPQWWOMYHFOHGKLRECB7NE5VBEGA, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.037833743 (117h46m32.908916778s) [noise=true , address=145.79.227.22:4509 , piece_id=62EZ3JKY7O5VZEGSATQYYBLKR46ASNJUX3H7YIYAIXAV4WYF26JQ, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.038339726 (117h46m32.909422772s) [noise=true , address=145.79.227.10:4401 , piece_id=S4DH6HYJ5IUWJ5BTURVW7OTDWFS3O3UNCZMSTUK4QZ4OCNKTH6MA, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.038758464 (117h46m32.909841509s) [noise=true , address=145.79.227.40:4049 , piece_id=FVAQGZN6T3VF2XVZB4PCEYGS3PU3XGLBSN4FLFQTYIEC5HDTC3OQ, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.039163404 (117h46m32.910246449s) [noise=true , address=77.90.31.17:4771 , piece_id=X57ZVYWUTGUUKAOIRL2FL5CHXAN5SQYLX6EJBQIKL4FZGHWGPRQA, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.039588294 (117h46m32.910671339s) [noise=true , address=145.79.227.39:4014 , piece_id=HO5AOQJRC774MOI5S6J7MOHIYV4L72XC22HLQCHBLYP3QYIA2DTQ, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.040016559 (117h46m32.911099594s) [noise=true , address=145.79.227.43:4159 , piece_id=5KI6NRQSBBPKKUN65NU5LTY32YJ65K2OKZDJQEVQX465WNQKNHHQ, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.040450956 (117h46m32.911534001s) [noise=true , address=77.90.31.10:4518 , piece_id=XCU5D2A36LBTDDYTILXWUMMANHVDCW2KMEAB3HUCDU5UONZLULBQ, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.040890654 (117h46m32.911973689s) [noise=true , address=145.79.227.22:4510 , piece_id=ALCQVB6CUD7JAJ7I2NHADFKGHOSFVBKCGR2KHZWHPMYKK3VV4T5A, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.041348756 (117h46m32.912431801s) [noise=true , address=45.91.116.11:4119 , piece_id=S5KLL473HCDKQBM5NVTVNLLSYGMXFFTFDFQX7D5SITM2MQO43K6Q, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.041761201 (117h46m32.912844236s) [noise=true , address=145.79.227.41:4087 , piece_id=OQCN6TONOCOR2EH7THCYREB7LRVYOMK55VPTAWK4NMADKYYYPYLA, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.042078305 (117h46m32.91316134s) [noise=true , address=145.79.227.7:4295 , piece_id=IPMI23BRBMF4MTC6EJJEZWGW7AMFPLGUGMXEFCF2QZASWEUFIH5Q, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.042443110 (117h46m32.913526155s) [noise=true , address=145.79.227.32:4835 , piece_id=4X2XQSZAB6KZPDY5G4VFDFZCL744RAL2IYUYMU7UWFMEII5D225A, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.042561475 (117h46m32.91364452s) [noise=true , address=145.79.227.31:4797 , piece_id=WOGJMBDWY42R5T5T5M2NR4AF3NFZ6CAYOI2O5RRL3JYPCAB3FFZA, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:35.033962808 (117h46m34.905045853s) [upload=102799]: Next returned nil segment Dec 10 13:15:35.033993557 (117h46m34.905076612s) [upload=102799]: Waiting for error group managing segments... Dec 10 13:15:35.355027663 (117h46m35.226110708s) [noise=true , address=77.90.31.10:4518 , piece_id=XCU5D2A36LBTDDYTILXWUMMANHVDCW2KMEAB3HUCDU5UONZLULBQ, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:35.395849857 (117h46m35.266932902s) [noise=true , address=77.90.31.17:4771 , piece_id=X57ZVYWUTGUUKAOIRL2FL5CHXAN5SQYLX6EJBQIKL4FZGHWGPRQA, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:35.939806675 (117h46m35.810889721s) [noise=true , address=145.79.227.31:4797 , piece_id=WOGJMBDWY42R5T5T5M2NR4AF3NFZ6CAYOI2O5RRL3JYPCAB3FFZA, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:35.944824321 (117h46m35.815907356s) [noise=true , address=145.79.227.7:4295 , piece_id=IPMI23BRBMF4MTC6EJJEZWGW7AMFPLGUGMXEFCF2QZASWEUFIH5Q, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:35.966925235 (117h46m35.83800827s) [noise=true , address=145.79.227.41:4087 , piece_id=OQCN6TONOCOR2EH7THCYREB7LRVYOMK55VPTAWK4NMADKYYYPYLA, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:36.183487725 (117h46m36.05457076s) [noise=true , address=145.79.227.29:4943 , piece_id=I5NHKVGJKAG37VAV53QOQ2ZYSDF6SGEXTTG7DEAAG5CC2DO7SVVQ, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:36.184533436 (117h46m36.055616471s) [noise=true , address=145.79.227.22:4509 , piece_id=62EZ3JKY7O5VZEGSATQYYBLKR46ASNJUX3H7YIYAIXAV4WYF26JQ, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:36.188784180 (117h46m36.059867215s) [noise=true , address=145.79.227.6:4259 , piece_id=RQCMHQGVQGAWFBJ6AAHMVON7SLBV7XPYR4D6XRI6DJEJVJ7AFGRQ, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:36.190871565 (117h46m36.0619546s) [noise=true , address=145.79.227.22:4510 , piece_id=ALCQVB6CUD7JAJ7I2NHADFKGHOSFVBKCGR2KHZWHPMYKK3VV4T5A, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:36.234924680 (117h46m36.106007725s) [noise=true , address=145.79.227.39:4014 , piece_id=HO5AOQJRC774MOI5S6J7MOHIYV4L72XC22HLQCHBLYP3QYIA2DTQ, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:37.136102891 (117h46m37.007185926s) [noise=true , address=145.79.227.32:4835 , piece_id=4X2XQSZAB6KZPDY5G4VFDFZCL744RAL2IYUYMU7UWFMEII5D225A, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:37.155928563 (117h46m37.027011618s) [noise=true , address=145.79.227.43:4159 , piece_id=5KI6NRQSBBPKKUN65NU5LTY32YJ65K2OKZDJQEVQX465WNQKNHHQ, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:37.199763735 (117h46m37.0708468s) [noise=true , address=145.79.227.10:4401 , piece_id=S4DH6HYJ5IUWJ5BTURVW7OTDWFS3O3UNCZMSTUK4QZ4OCNKTH6MA, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:37.199789233 (117h46m37.070872279s) [seg_pos={0 0}, upload=102799]: Segment reached optimal threshold of 13 pieces. Dec 10 13:15:37.200041654 (117h46m37.071124689s) [noise=true , address=145.79.227.26:4654 , piece_id=TRV36WT3M3K5LIJB3LCMLG6OFPQWWOMYHFOHGKLRECB7NE5VBEGA, seg_pos={0 0}, upload=102799]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KhBWMerZe9tUmzdVEYkBb6NmJ25dGP26Ea7CLjNJT5gdAeiah): protocol: write tcp 188.166.225.170:53216->145.79.227.26:4654: use of closed network connection; write tcp 188.166.225.170:53216->145.79.227.26:4654: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53216->145.79.227.26:4654: use of closed network connection Dec 10 13:15:37.200055591 (117h46m37.071138656s) [noise=true , address=145.79.227.26:4654 , piece_id=TRV36WT3M3K5LIJB3LCMLG6OFPQWWOMYHFOHGKLRECB7NE5VBEGA, seg_pos={0 0}, upload=102799]: Long tail cancellation due to optimal threshold Dec 10 13:15:37.200156243 (117h46m37.071239288s) [noise=true , address=145.79.227.40:4049 , piece_id=FVAQGZN6T3VF2XVZB4PCEYGS3PU3XGLBSN4FLFQTYIEC5HDTC3OQ, seg_pos={0 0}, upload=102799]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J): protocol: write tcp 188.166.225.170:55280->145.79.227.40:4049: use of closed network connection; write tcp 188.166.225.170:55280->145.79.227.40:4049: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55280->145.79.227.40:4049: use of closed network connection Dec 10 13:15:37.200184998 (117h46m37.071268053s) [noise=true , address=145.79.227.40:4049 , piece_id=FVAQGZN6T3VF2XVZB4PCEYGS3PU3XGLBSN4FLFQTYIEC5HDTC3OQ, seg_pos={0 0}, upload=102799]: Long tail cancellation due to optimal threshold Dec 10 13:15:37.200373426 (117h46m37.071456461s) [noise=true , address=45.91.116.11:4119 , piece_id=S5KLL473HCDKQBM5NVTVNLLSYGMXFFTFDFQX7D5SITM2MQO43K6Q, seg_pos={0 0}, upload=102799]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: expected piece hash; context canceled; EOF Dec 10 13:15:37.200381421 (117h46m37.071464466s) [noise=true , address=45.91.116.11:4119 , piece_id=S5KLL473HCDKQBM5NVTVNLLSYGMXFFTFDFQX7D5SITM2MQO43K6Q, seg_pos={0 0}, upload=102799]: Long tail cancellation due to optimal threshold Dec 10 13:15:37.200402562 (117h46m37.071485597s) [seg_pos={0 0}, upload=102799]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 10 13:15:37.200491951 (117h46m37.071574996s) [upload=102799]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 10 13:15:37.200500108 (117h46m37.071583153s) [upload=102799]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 10 13:15:37.624562388 (117h46m37.495645534s) [upload=102799]: Done uploading segments... Dec 10 13:15:37.624584852 (117h46m37.495667887s) [upload=102799]: Upload finished. err: <nil> Duration: 14.089086504s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [........X...........................................X..X.X..] duration: 15.770798714s trace: https://jaeger.dev.storj.io/trace/4e7e66895306662b FAIL (2025-12-10 13:27:54 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 15.770823681s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 52 out of 60 is failed (threshold=6) [..X.XXXXXXXXXXXXXXXXXXXXXXXX..X.XXXX.XXXXXXXXXXXX.XXXXXXXXXX] duration: 5.00082333s trace: https://jaeger.dev.storj.io/trace/9fe24f9ea0514d1 FAIL (2025-12-10 13:27:40 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000848448s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 9.786932777s trace: https://jaeger.dev.storj.io/trace/a0b5a94505b001 FAIL (2025-12-10 13:27:37 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:36068->45.38.24.2:443: read: connection reset by peer Duration: 9.786959619s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.....X...X.................................X................] duration: 8.444095123s trace: https://jaeger.dev.storj.io/trace/19144dc231ae8183 FAIL (2025-12-10 13:26:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ts... Dec 10 13:15:32.674883550 (117h46m32.545966595s) [upload=102799]: Got next segment. Inline: false Dec 10 13:15:32.674940880 (117h46m32.546023915s) [upload=102799]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 10 13:15:32.674946670 (117h46m32.546029706s) [upload=102799]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 10 13:15:33.036416584 (117h46m32.907499629s) [seg_pos={0 0}, upload=102799]: Begin upload segment... Dec 10 13:15:33.036472831 (117h46m32.907555876s) [seg_pos={0 0}, upload=102799]: Done begin upload segment. Dec 10 13:15:33.036565578 (117h46m32.907648613s) [noise=true , address=145.79.227.29:4943 , piece_id=I5NHKVGJKAG37VAV53QOQ2ZYSDF6SGEXTTG7DEAAG5CC2DO7SVVQ, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.036606606 (117h46m32.907689651s) [noise=true , address=145.79.227.6:4259 , piece_id=RQCMHQGVQGAWFBJ6AAHMVON7SLBV7XPYR4D6XRI6DJEJVJ7AFGRQ, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.037385819 (117h46m32.908468865s) [noise=true , address=145.79.227.26:4654 , piece_id=TRV36WT3M3K5LIJB3LCMLG6OFPQWWOMYHFOHGKLRECB7NE5VBEGA, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.037833743 (117h46m32.908916778s) [noise=true , address=145.79.227.22:4509 , piece_id=62EZ3JKY7O5VZEGSATQYYBLKR46ASNJUX3H7YIYAIXAV4WYF26JQ, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.038339726 (117h46m32.909422772s) [noise=true , address=145.79.227.10:4401 , piece_id=S4DH6HYJ5IUWJ5BTURVW7OTDWFS3O3UNCZMSTUK4QZ4OCNKTH6MA, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.038758464 (117h46m32.909841509s) [noise=true , address=145.79.227.40:4049 , piece_id=FVAQGZN6T3VF2XVZB4PCEYGS3PU3XGLBSN4FLFQTYIEC5HDTC3OQ, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.039163404 (117h46m32.910246449s) [noise=true , address=77.90.31.17:4771 , piece_id=X57ZVYWUTGUUKAOIRL2FL5CHXAN5SQYLX6EJBQIKL4FZGHWGPRQA, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.039588294 (117h46m32.910671339s) [noise=true , address=145.79.227.39:4014 , piece_id=HO5AOQJRC774MOI5S6J7MOHIYV4L72XC22HLQCHBLYP3QYIA2DTQ, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.040016559 (117h46m32.911099594s) [noise=true , address=145.79.227.43:4159 , piece_id=5KI6NRQSBBPKKUN65NU5LTY32YJ65K2OKZDJQEVQX465WNQKNHHQ, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.040450956 (117h46m32.911534001s) [noise=true , address=77.90.31.10:4518 , piece_id=XCU5D2A36LBTDDYTILXWUMMANHVDCW2KMEAB3HUCDU5UONZLULBQ, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.040890654 (117h46m32.911973689s) [noise=true , address=145.79.227.22:4510 , piece_id=ALCQVB6CUD7JAJ7I2NHADFKGHOSFVBKCGR2KHZWHPMYKK3VV4T5A, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.041348756 (117h46m32.912431801s) [noise=true , address=45.91.116.11:4119 , piece_id=S5KLL473HCDKQBM5NVTVNLLSYGMXFFTFDFQX7D5SITM2MQO43K6Q, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.041761201 (117h46m32.912844236s) [noise=true , address=145.79.227.41:4087 , piece_id=OQCN6TONOCOR2EH7THCYREB7LRVYOMK55VPTAWK4NMADKYYYPYLA, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.042078305 (117h46m32.91316134s) [noise=true , address=145.79.227.7:4295 , piece_id=IPMI23BRBMF4MTC6EJJEZWGW7AMFPLGUGMXEFCF2QZASWEUFIH5Q, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.042443110 (117h46m32.913526155s) [noise=true , address=145.79.227.32:4835 , piece_id=4X2XQSZAB6KZPDY5G4VFDFZCL744RAL2IYUYMU7UWFMEII5D225A, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.042561475 (117h46m32.91364452s) [noise=true , address=145.79.227.31:4797 , piece_id=WOGJMBDWY42R5T5T5M2NR4AF3NFZ6CAYOI2O5RRL3JYPCAB3FFZA, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:35.033962808 (117h46m34.905045853s) [upload=102799]: Next returned nil segment Dec 10 13:15:35.033993557 (117h46m34.905076612s) [upload=102799]: Waiting for error group managing segments... Dec 10 13:15:35.355027663 (117h46m35.226110708s) [noise=true , address=77.90.31.10:4518 , piece_id=XCU5D2A36LBTDDYTILXWUMMANHVDCW2KMEAB3HUCDU5UONZLULBQ, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:35.395849857 (117h46m35.266932902s) [noise=true , address=77.90.31.17:4771 , piece_id=X57ZVYWUTGUUKAOIRL2FL5CHXAN5SQYLX6EJBQIKL4FZGHWGPRQA, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:35.939806675 (117h46m35.810889721s) [noise=true , address=145.79.227.31:4797 , piece_id=WOGJMBDWY42R5T5T5M2NR4AF3NFZ6CAYOI2O5RRL3JYPCAB3FFZA, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:35.944824321 (117h46m35.815907356s) [noise=true , address=145.79.227.7:4295 , piece_id=IPMI23BRBMF4MTC6EJJEZWGW7AMFPLGUGMXEFCF2QZASWEUFIH5Q, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:35.966925235 (117h46m35.83800827s) [noise=true , address=145.79.227.41:4087 , piece_id=OQCN6TONOCOR2EH7THCYREB7LRVYOMK55VPTAWK4NMADKYYYPYLA, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:36.183487725 (117h46m36.05457076s) [noise=true , address=145.79.227.29:4943 , piece_id=I5NHKVGJKAG37VAV53QOQ2ZYSDF6SGEXTTG7DEAAG5CC2DO7SVVQ, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:36.184533436 (117h46m36.055616471s) [noise=true , address=145.79.227.22:4509 , piece_id=62EZ3JKY7O5VZEGSATQYYBLKR46ASNJUX3H7YIYAIXAV4WYF26JQ, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:36.188784180 (117h46m36.059867215s) [noise=true , address=145.79.227.6:4259 , piece_id=RQCMHQGVQGAWFBJ6AAHMVON7SLBV7XPYR4D6XRI6DJEJVJ7AFGRQ, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:36.190871565 (117h46m36.0619546s) [noise=true , address=145.79.227.22:4510 , piece_id=ALCQVB6CUD7JAJ7I2NHADFKGHOSFVBKCGR2KHZWHPMYKK3VV4T5A, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:36.234924680 (117h46m36.106007725s) [noise=true , address=145.79.227.39:4014 , piece_id=HO5AOQJRC774MOI5S6J7MOHIYV4L72XC22HLQCHBLYP3QYIA2DTQ, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:37.136102891 (117h46m37.007185926s) [noise=true , address=145.79.227.32:4835 , piece_id=4X2XQSZAB6KZPDY5G4VFDFZCL744RAL2IYUYMU7UWFMEII5D225A, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:37.155928563 (117h46m37.027011618s) [noise=true , address=145.79.227.43:4159 , piece_id=5KI6NRQSBBPKKUN65NU5LTY32YJ65K2OKZDJQEVQX465WNQKNHHQ, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:37.199763735 (117h46m37.0708468s) [noise=true , address=145.79.227.10:4401 , piece_id=S4DH6HYJ5IUWJ5BTURVW7OTDWFS3O3UNCZMSTUK4QZ4OCNKTH6MA, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:37.199789233 (117h46m37.070872279s) [seg_pos={0 0}, upload=102799]: Segment reached optimal threshold of 13 pieces. Dec 10 13:15:37.200041654 (117h46m37.071124689s) [noise=true , address=145.79.227.26:4654 , piece_id=TRV36WT3M3K5LIJB3LCMLG6OFPQWWOMYHFOHGKLRECB7NE5VBEGA, seg_pos={0 0}, upload=102799]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KhBWMerZe9tUmzdVEYkBb6NmJ25dGP26Ea7CLjNJT5gdAeiah): protocol: write tcp 188.166.225.170:53216->145.79.227.26:4654: use of closed network connection; write tcp 188.166.225.170:53216->145.79.227.26:4654: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53216->145.79.227.26:4654: use of closed network connection Dec 10 13:15:37.200055591 (117h46m37.071138656s) [noise=true , address=145.79.227.26:4654 , piece_id=TRV36WT3M3K5LIJB3LCMLG6OFPQWWOMYHFOHGKLRECB7NE5VBEGA, seg_pos={0 0}, upload=102799]: Long tail cancellation due to optimal threshold Dec 10 13:15:37.200156243 (117h46m37.071239288s) [noise=true , address=145.79.227.40:4049 , piece_id=FVAQGZN6T3VF2XVZB4PCEYGS3PU3XGLBSN4FLFQTYIEC5HDTC3OQ, seg_pos={0 0}, upload=102799]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J): protocol: write tcp 188.166.225.170:55280->145.79.227.40:4049: use of closed network connection; write tcp 188.166.225.170:55280->145.79.227.40:4049: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55280->145.79.227.40:4049: use of closed network connection Dec 10 13:15:37.200184998 (117h46m37.071268053s) [noise=true , address=145.79.227.40:4049 , piece_id=FVAQGZN6T3VF2XVZB4PCEYGS3PU3XGLBSN4FLFQTYIEC5HDTC3OQ, seg_pos={0 0}, upload=102799]: Long tail cancellation due to optimal threshold Dec 10 13:15:37.200373426 (117h46m37.071456461s) [noise=true , address=45.91.116.11:4119 , piece_id=S5KLL473HCDKQBM5NVTVNLLSYGMXFFTFDFQX7D5SITM2MQO43K6Q, seg_pos={0 0}, upload=102799]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: expected piece hash; context canceled; EOF Dec 10 13:15:37.200381421 (117h46m37.071464466s) [noise=true , address=45.91.116.11:4119 , piece_id=S5KLL473HCDKQBM5NVTVNLLSYGMXFFTFDFQX7D5SITM2MQO43K6Q, seg_pos={0 0}, upload=102799]: Long tail cancellation due to optimal threshold Dec 10 13:15:37.200402562 (117h46m37.071485597s) [seg_pos={0 0}, upload=102799]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 10 13:15:37.200491951 (117h46m37.071574996s) [upload=102799]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 10 13:15:37.200500108 (117h46m37.071583153s) [upload=102799]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 10 13:15:37.624562388 (117h46m37.495645534s) [upload=102799]: Done uploading segments... Dec 10 13:15:37.624584852 (117h46m37.495667887s) [upload=102799]: Upload finished. err: <nil> Duration: 8.444126863s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.........X...........................................X..X.X.] duration: 11.808590479s trace: https://jaeger.dev.storj.io/trace/4e7e66895306662b FAIL (2025-12-10 13:26:42 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 11.808623802s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 52 out of 60 is failed (threshold=6) [X..X.XXXXXXXXXXXXXXXXXXXXXXXX..X.XXXX.XXXXXXXXXXXX.XXXXXXXXX] duration: 5.001815905s trace: https://jaeger.dev.storj.io/trace/56579467ceeca9bc FAIL (2025-12-10 13:26:39 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001838217s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [......X...X.................................X...............] duration: 12.856045076s trace: https://jaeger.dev.storj.io/trace/19144dc231ae8183 FAIL (2025-12-10 13:26:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ts... Dec 10 13:15:32.674883550 (117h46m32.545966595s) [upload=102799]: Got next segment. Inline: false Dec 10 13:15:32.674940880 (117h46m32.546023915s) [upload=102799]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 10 13:15:32.674946670 (117h46m32.546029706s) [upload=102799]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 10 13:15:33.036416584 (117h46m32.907499629s) [seg_pos={0 0}, upload=102799]: Begin upload segment... Dec 10 13:15:33.036472831 (117h46m32.907555876s) [seg_pos={0 0}, upload=102799]: Done begin upload segment. Dec 10 13:15:33.036565578 (117h46m32.907648613s) [noise=true , address=145.79.227.29:4943 , piece_id=I5NHKVGJKAG37VAV53QOQ2ZYSDF6SGEXTTG7DEAAG5CC2DO7SVVQ, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.036606606 (117h46m32.907689651s) [noise=true , address=145.79.227.6:4259 , piece_id=RQCMHQGVQGAWFBJ6AAHMVON7SLBV7XPYR4D6XRI6DJEJVJ7AFGRQ, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.037385819 (117h46m32.908468865s) [noise=true , address=145.79.227.26:4654 , piece_id=TRV36WT3M3K5LIJB3LCMLG6OFPQWWOMYHFOHGKLRECB7NE5VBEGA, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.037833743 (117h46m32.908916778s) [noise=true , address=145.79.227.22:4509 , piece_id=62EZ3JKY7O5VZEGSATQYYBLKR46ASNJUX3H7YIYAIXAV4WYF26JQ, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.038339726 (117h46m32.909422772s) [noise=true , address=145.79.227.10:4401 , piece_id=S4DH6HYJ5IUWJ5BTURVW7OTDWFS3O3UNCZMSTUK4QZ4OCNKTH6MA, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.038758464 (117h46m32.909841509s) [noise=true , address=145.79.227.40:4049 , piece_id=FVAQGZN6T3VF2XVZB4PCEYGS3PU3XGLBSN4FLFQTYIEC5HDTC3OQ, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.039163404 (117h46m32.910246449s) [noise=true , address=77.90.31.17:4771 , piece_id=X57ZVYWUTGUUKAOIRL2FL5CHXAN5SQYLX6EJBQIKL4FZGHWGPRQA, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.039588294 (117h46m32.910671339s) [noise=true , address=145.79.227.39:4014 , piece_id=HO5AOQJRC774MOI5S6J7MOHIYV4L72XC22HLQCHBLYP3QYIA2DTQ, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.040016559 (117h46m32.911099594s) [noise=true , address=145.79.227.43:4159 , piece_id=5KI6NRQSBBPKKUN65NU5LTY32YJ65K2OKZDJQEVQX465WNQKNHHQ, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.040450956 (117h46m32.911534001s) [noise=true , address=77.90.31.10:4518 , piece_id=XCU5D2A36LBTDDYTILXWUMMANHVDCW2KMEAB3HUCDU5UONZLULBQ, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.040890654 (117h46m32.911973689s) [noise=true , address=145.79.227.22:4510 , piece_id=ALCQVB6CUD7JAJ7I2NHADFKGHOSFVBKCGR2KHZWHPMYKK3VV4T5A, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.041348756 (117h46m32.912431801s) [noise=true , address=45.91.116.11:4119 , piece_id=S5KLL473HCDKQBM5NVTVNLLSYGMXFFTFDFQX7D5SITM2MQO43K6Q, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.041761201 (117h46m32.912844236s) [noise=true , address=145.79.227.41:4087 , piece_id=OQCN6TONOCOR2EH7THCYREB7LRVYOMK55VPTAWK4NMADKYYYPYLA, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.042078305 (117h46m32.91316134s) [noise=true , address=145.79.227.7:4295 , piece_id=IPMI23BRBMF4MTC6EJJEZWGW7AMFPLGUGMXEFCF2QZASWEUFIH5Q, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.042443110 (117h46m32.913526155s) [noise=true , address=145.79.227.32:4835 , piece_id=4X2XQSZAB6KZPDY5G4VFDFZCL744RAL2IYUYMU7UWFMEII5D225A, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:33.042561475 (117h46m32.91364452s) [noise=true , address=145.79.227.31:4797 , piece_id=WOGJMBDWY42R5T5T5M2NR4AF3NFZ6CAYOI2O5RRL3JYPCAB3FFZA, seg_pos={0 0}, upload=102799]: Uploading piece... Dec 10 13:15:35.033962808 (117h46m34.905045853s) [upload=102799]: Next returned nil segment Dec 10 13:15:35.033993557 (117h46m34.905076612s) [upload=102799]: Waiting for error group managing segments... Dec 10 13:15:35.355027663 (117h46m35.226110708s) [noise=true , address=77.90.31.10:4518 , piece_id=XCU5D2A36LBTDDYTILXWUMMANHVDCW2KMEAB3HUCDU5UONZLULBQ, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:35.395849857 (117h46m35.266932902s) [noise=true , address=77.90.31.17:4771 , piece_id=X57ZVYWUTGUUKAOIRL2FL5CHXAN5SQYLX6EJBQIKL4FZGHWGPRQA, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:35.939806675 (117h46m35.810889721s) [noise=true , address=145.79.227.31:4797 , piece_id=WOGJMBDWY42R5T5T5M2NR4AF3NFZ6CAYOI2O5RRL3JYPCAB3FFZA, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:35.944824321 (117h46m35.815907356s) [noise=true , address=145.79.227.7:4295 , piece_id=IPMI23BRBMF4MTC6EJJEZWGW7AMFPLGUGMXEFCF2QZASWEUFIH5Q, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:35.966925235 (117h46m35.83800827s) [noise=true , address=145.79.227.41:4087 , piece_id=OQCN6TONOCOR2EH7THCYREB7LRVYOMK55VPTAWK4NMADKYYYPYLA, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:36.183487725 (117h46m36.05457076s) [noise=true , address=145.79.227.29:4943 , piece_id=I5NHKVGJKAG37VAV53QOQ2ZYSDF6SGEXTTG7DEAAG5CC2DO7SVVQ, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:36.184533436 (117h46m36.055616471s) [noise=true , address=145.79.227.22:4509 , piece_id=62EZ3JKY7O5VZEGSATQYYBLKR46ASNJUX3H7YIYAIXAV4WYF26JQ, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:36.188784180 (117h46m36.059867215s) [noise=true , address=145.79.227.6:4259 , piece_id=RQCMHQGVQGAWFBJ6AAHMVON7SLBV7XPYR4D6XRI6DJEJVJ7AFGRQ, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:36.190871565 (117h46m36.0619546s) [noise=true , address=145.79.227.22:4510 , piece_id=ALCQVB6CUD7JAJ7I2NHADFKGHOSFVBKCGR2KHZWHPMYKK3VV4T5A, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:36.234924680 (117h46m36.106007725s) [noise=true , address=145.79.227.39:4014 , piece_id=HO5AOQJRC774MOI5S6J7MOHIYV4L72XC22HLQCHBLYP3QYIA2DTQ, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:37.136102891 (117h46m37.007185926s) [noise=true , address=145.79.227.32:4835 , piece_id=4X2XQSZAB6KZPDY5G4VFDFZCL744RAL2IYUYMU7UWFMEII5D225A, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:37.155928563 (117h46m37.027011618s) [noise=true , address=145.79.227.43:4159 , piece_id=5KI6NRQSBBPKKUN65NU5LTY32YJ65K2OKZDJQEVQX465WNQKNHHQ, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:37.199763735 (117h46m37.0708468s) [noise=true , address=145.79.227.10:4401 , piece_id=S4DH6HYJ5IUWJ5BTURVW7OTDWFS3O3UNCZMSTUK4QZ4OCNKTH6MA, seg_pos={0 0}, upload=102799]: Done uploading piece. err: <nil> Dec 10 13:15:37.199789233 (117h46m37.070872279s) [seg_pos={0 0}, upload=102799]: Segment reached optimal threshold of 13 pieces. Dec 10 13:15:37.200041654 (117h46m37.071124689s) [noise=true , address=145.79.227.26:4654 , piece_id=TRV36WT3M3K5LIJB3LCMLG6OFPQWWOMYHFOHGKLRECB7NE5VBEGA, seg_pos={0 0}, upload=102799]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KhBWMerZe9tUmzdVEYkBb6NmJ25dGP26Ea7CLjNJT5gdAeiah): protocol: write tcp 188.166.225.170:53216->145.79.227.26:4654: use of closed network connection; write tcp 188.166.225.170:53216->145.79.227.26:4654: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53216->145.79.227.26:4654: use of closed network connection Dec 10 13:15:37.200055591 (117h46m37.071138656s) [noise=true , address=145.79.227.26:4654 , piece_id=TRV36WT3M3K5LIJB3LCMLG6OFPQWWOMYHFOHGKLRECB7NE5VBEGA, seg_pos={0 0}, upload=102799]: Long tail cancellation due to optimal threshold Dec 10 13:15:37.200156243 (117h46m37.071239288s) [noise=true , address=145.79.227.40:4049 , piece_id=FVAQGZN6T3VF2XVZB4PCEYGS3PU3XGLBSN4FLFQTYIEC5HDTC3OQ, seg_pos={0 0}, upload=102799]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J): protocol: write tcp 188.166.225.170:55280->145.79.227.40:4049: use of closed network connection; write tcp 188.166.225.170:55280->145.79.227.40:4049: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55280->145.79.227.40:4049: use of closed network connection Dec 10 13:15:37.200184998 (117h46m37.071268053s) [noise=true , address=145.79.227.40:4049 , piece_id=FVAQGZN6T3VF2XVZB4PCEYGS3PU3XGLBSN4FLFQTYIEC5HDTC3OQ, seg_pos={0 0}, upload=102799]: Long tail cancellation due to optimal threshold Dec 10 13:15:37.200373426 (117h46m37.071456461s) [noise=true , address=45.91.116.11:4119 , piece_id=S5KLL473HCDKQBM5NVTVNLLSYGMXFFTFDFQX7D5SITM2MQO43K6Q, seg_pos={0 0}, upload=102799]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: expected piece hash; context canceled; EOF Dec 10 13:15:37.200381421 (117h46m37.071464466s) [noise=true , address=45.91.116.11:4119 , piece_id=S5KLL473HCDKQBM5NVTVNLLSYGMXFFTFDFQX7D5SITM2MQO43K6Q, seg_pos={0 0}, upload=102799]: Long tail cancellation due to optimal threshold Dec 10 13:15:37.200402562 (117h46m37.071485597s) [seg_pos={0 0}, upload=102799]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 10 13:15:37.200491951 (117h46m37.071574996s) [upload=102799]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 10 13:15:37.200500108 (117h46m37.071583153s) [upload=102799]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 10 13:15:37.624562388 (117h46m37.495645534s) [upload=102799]: Done uploading segments... Dec 10 13:15:37.624584852 (117h46m37.495667887s) [upload=102799]: Upload finished. err: <nil> Duration: 12.856075044s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 6.089561348s trace: https://jaeger.dev.storj.io/trace/a0b5a94505b001 FAIL (2025-12-10 13:26:22 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:36068->45.38.24.2:443: read: connection reset by peer Duration: 6.089606084s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [..........X...........................................X..X.X] duration: 30.000687693s trace: https://jaeger.dev.storj.io/trace/4e7e66895306662b FAIL (2025-12-10 13:25:47 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.000699175s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 52 out of 60 is failed (threshold=6) [XX..X.XXXXXXXXXXXXXXXXXXXXXXXX..X.XXXX.XXXXXXXXXXXX.XXXXXXXX] duration: 5.001015322s trace: https://jaeger.dev.storj.io/trace/fa8d38721ad2988 FAIL (2025-12-10 13:25:47 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001041622s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 4.78396837s trace: https://jaeger.dev.storj.io/trace/a0b5a94505b001 FAIL (2025-12-10 13:25:45 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:36068->45.38.24.2:443: read: connection reset by peer Duration: 4.783998528s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 4.68203997s trace: https://jaeger.dev.storj.io/trace/a0b5a94505b001 FAIL (2025-12-10 13:25:27 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:36068->45.38.24.2:443: read: connection reset by peer Duration: 4.682069535s edge gateway us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)