Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 6.186297096s trace: https://jaeger.dev.storj.io/trace/3c77028f820d12f9 FAIL (2025-11-16 14:27:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, Piece ID: UKP3MHXBSALCWFIHCTSR5JFFXEWWAWGDS4JYUYS2JEREQQKTPJSA) context deadline exceeded; piecestore: (Node ID: 12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, Piece ID: DNT5QVAMRO5OMBNVTDRRUKACMM7M5472LD5E4ASKYIBKBNWYAVRA) context deadline exceeded; piecestore: (Node ID: 17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, Piece ID: DPLALMJC4JPC5D7BOK5SOSDFDJFDDPGSJWP4XDDQE22AMFNAKATQ) context deadline exceeded ======LOGS====== gment. Inline: false Nov 16 13:42:52.666975419 (173h38m51.21844588s) [upload=152907]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 16 13:42:52.666980539 (173h38m51.21845099s) [upload=152907]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 16 13:42:53.024947049 (173h38m51.5764175s) [seg_pos={0 0}, upload=152907]: Begin upload segment... Nov 16 13:42:53.025007074 (173h38m51.576477535s) [seg_pos={0 0}, upload=152907]: Done begin upload segment. Nov 16 13:42:53.025087707 (173h38m51.576558158s) [noise=true , address=77.90.31.14:4662 , piece_id=S5PPMCKIKSOCLUJ4CEH7ZDW3TVUZ7GTFFEIPCQK6FX25UIOUQHHQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.025098377 (173h38m51.576568828s) [noise=true , address=145.79.227.31:4797 , piece_id=UKP3MHXBSALCWFIHCTSR5JFFXEWWAWGDS4JYUYS2JEREQQKTPJSA, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.025961451 (173h38m51.577431912s) [noise=true , address=45.91.116.6:4974 , piece_id=ZTS3H7E6A3VJECLRRKRKSVX5PH3VSH7F2HAUPQSPH56NCI6RTFBQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.026101888 (173h38m51.577572339s) [noise=true , address=145.79.227.10:4402 , piece_id=DNT5QVAMRO5OMBNVTDRRUKACMM7M5472LD5E4ASKYIBKBNWYAVRA, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.026397721 (173h38m51.577868182s) [noise=true , address=145.79.227.11:4732 , piece_id=GR6SASHNUTHAEOUPK7DWKVPHRGMG72FQMF4RINJ5NTIQQGFCVZZQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.026543650 (173h38m51.578014111s) [noise=true , address=145.79.227.20:4437 , piece_id=G5P6OYXVAAQDZKTPPHDNZLMYTT6L2TFWKH7N4F7L6742BWHOQGBQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.026887915 (173h38m51.578358366s) [noise=true , address=45.91.116.14:4189 , piece_id=NUMS6Q3F2OUAQ2PSKE2CZK2WKRQLZUZPWVBA3UA6UNKDQACPYSJQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.027039624 (173h38m51.578510085s) [noise=true , address=145.79.227.27:4691 , piece_id=NSX5S2D7ZSA5I57B4EAQEQXEVUMPKNMRKRV3RHWGWJ2DN7SWUPGQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.027374482 (173h38m51.578844973s) [noise=true , address=145.79.227.31:4798 , piece_id=DPLALMJC4JPC5D7BOK5SOSDFDJFDDPGSJWP4XDDQE22AMFNAKATQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.027472337 (173h38m51.578942798s) [noise=true , address=45.91.116.7:4010 , piece_id=EO7IEOCWEJAMZZU4BLIB5ALFACWTWZE3QCXU3FUY4FKC4P444ORA, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.027920821 (173h38m51.579391283s) [noise=true , address=145.79.227.23:4547 , piece_id=P4WX2ZDHV2Z6BFMF6IACIYW4DXKYFVSULUTF7EK2EMRO5OA5X5RQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.028194272 (173h38m51.579664743s) [noise=true , address=145.79.227.32:4834 , piece_id=CBK2UQ252EPFQDTOYS2X5WAUTJN4L2DTHGIOJXFZXWBOLGPRETXQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.028430262 (173h38m51.579900723s) [noise=true , address=145.79.227.9:4366 , piece_id=YFWF2YL2YPYVMP5DHJKRUGVUUS2ALG4DRJ3HUZCDZM4N3N676FNQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.029687726 (173h38m51.581158177s) [noise=true , address=145.79.227.20:4438 , piece_id=6HY2HDNJAZ552ZKXFJ7VNTQNBAQU5ZQLNKAG2NZZWZQHN6EEDG6Q, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.031784128 (173h38m51.583254589s) [noise=true , address=45.91.116.5:4298 , piece_id=ARGPAZE6CTMM6UXSMPSVHDBEQWKVGKY7WXOVLPB7UH3JZYMDB7JQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.034100830 (173h38m51.585571291s) [noise=true , address=77.90.31.10:4518 , piece_id=MSKQ66KE4WANAE4AWOD2IKY3PQN2YGZ5N4W3NXGKXSOLE7B6H6WQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:54.850200282 (173h38m53.401670733s) [upload=152907]: Next returned nil segment Nov 16 13:42:54.850219118 (173h38m53.401689569s) [upload=152907]: Waiting for error group managing segments... Nov 16 13:42:55.154333032 (173h38m53.705803493s) [noise=true , address=45.91.116.7:4010 , piece_id=EO7IEOCWEJAMZZU4BLIB5ALFACWTWZE3QCXU3FUY4FKC4P444ORA, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:42:55.201689040 (173h38m53.753159501s) [noise=true , address=77.90.31.10:4518 , piece_id=MSKQ66KE4WANAE4AWOD2IKY3PQN2YGZ5N4W3NXGKXSOLE7B6H6WQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:42:55.232313345 (173h38m53.783783816s) [noise=true , address=45.91.116.5:4298 , piece_id=ARGPAZE6CTMM6UXSMPSVHDBEQWKVGKY7WXOVLPB7UH3JZYMDB7JQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:42:55.260281393 (173h38m53.811751854s) [noise=true , address=45.91.116.6:4974 , piece_id=ZTS3H7E6A3VJECLRRKRKSVX5PH3VSH7F2HAUPQSPH56NCI6RTFBQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:00.229127085 (173h38m58.780597546s) [noise=true , address=145.79.227.20:4438 , piece_id=6HY2HDNJAZ552ZKXFJ7VNTQNBAQU5ZQLNKAG2NZZWZQHN6EEDG6Q, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:00.358241119 (173h38m58.909711581s) [noise=true , address=145.79.227.32:4834 , piece_id=CBK2UQ252EPFQDTOYS2X5WAUTJN4L2DTHGIOJXFZXWBOLGPRETXQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:00.646958215 (173h38m59.198428676s) [noise=true , address=145.79.227.31:4798 , piece_id=DPLALMJC4JPC5D7BOK5SOSDFDJFDDPGSJWP4XDDQE22AMFNAKATQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:00.908824080 (173h38m59.460294541s) [noise=true , address=145.79.227.23:4547 , piece_id=P4WX2ZDHV2Z6BFMF6IACIYW4DXKYFVSULUTF7EK2EMRO5OA5X5RQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:01.155346527 (173h38m59.706816989s) [noise=true , address=45.91.116.14:4189 , piece_id=NUMS6Q3F2OUAQ2PSKE2CZK2WKRQLZUZPWVBA3UA6UNKDQACPYSJQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:01.839444794 (173h39m0.390915255s) [noise=true , address=145.79.227.11:4732 , piece_id=GR6SASHNUTHAEOUPK7DWKVPHRGMG72FQMF4RINJ5NTIQQGFCVZZQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:02.591249579 (173h39m1.14272004s) [noise=true , address=145.79.227.10:4402 , piece_id=DNT5QVAMRO5OMBNVTDRRUKACMM7M5472LD5E4ASKYIBKBNWYAVRA, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:02.616126740 (173h39m1.167597202s) [noise=true , address=145.79.227.31:4797 , piece_id=UKP3MHXBSALCWFIHCTSR5JFFXEWWAWGDS4JYUYS2JEREQQKTPJSA, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:02.734605482 (173h39m1.286075943s) [noise=true , address=145.79.227.27:4691 , piece_id=NSX5S2D7ZSA5I57B4EAQEQXEVUMPKNMRKRV3RHWGWJ2DN7SWUPGQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:02.734638004 (173h39m1.286108475s) [seg_pos={0 0}, upload=152907]: Segment reached optimal threshold of 13 pieces. Nov 16 13:43:02.735049748 (173h39m1.286520209s) [noise=true , address=145.79.227.9:4366 , piece_id=YFWF2YL2YPYVMP5DHJKRUGVUUS2ALG4DRJ3HUZCDZM4N3N676FNQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy): protocol: write tcp 188.166.225.170:60100->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:60100->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60100->145.79.227.9:4366: use of closed network connection Nov 16 13:43:02.735060949 (173h39m1.28653141s) [noise=true , address=145.79.227.9:4366 , piece_id=YFWF2YL2YPYVMP5DHJKRUGVUUS2ALG4DRJ3HUZCDZM4N3N676FNQ, seg_pos={0 0}, upload=152907]: Long tail cancellation due to optimal threshold Nov 16 13:43:02.735509193 (173h39m1.286979654s) [noise=true , address=77.90.31.14:4662 , piece_id=S5PPMCKIKSOCLUJ4CEH7ZDW3TVUZ7GTFFEIPCQK6FX25UIOUQHHQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): protocol: write tcp 188.166.225.170:50196->77.90.31.14:4662: use of closed network connection; write tcp 188.166.225.170:50196->77.90.31.14:4662: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50196->77.90.31.14:4662: use of closed network connection Nov 16 13:43:02.735518792 (173h39m1.286989253s) [noise=true , address=77.90.31.14:4662 , piece_id=S5PPMCKIKSOCLUJ4CEH7ZDW3TVUZ7GTFFEIPCQK6FX25UIOUQHHQ, seg_pos={0 0}, upload=152907]: Long tail cancellation due to optimal threshold Nov 16 13:43:02.736224335 (173h39m1.287694796s) [noise=true , address=145.79.227.20:4437 , piece_id=G5P6OYXVAAQDZKTPPHDNZLMYTT6L2TFWKH7N4F7L6742BWHOQGBQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV): protocol: read tcp 188.166.225.170:35300->145.79.227.20:4437: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 13:43:02.736239203 (173h39m1.287709654s) [noise=true , address=145.79.227.20:4437 , piece_id=G5P6OYXVAAQDZKTPPHDNZLMYTT6L2TFWKH7N4F7L6742BWHOQGBQ, seg_pos={0 0}, upload=152907]: Long tail cancellation due to optimal threshold Nov 16 13:43:02.736270833 (173h39m1.287741294s) [seg_pos={0 0}, upload=152907]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 16 13:43:02.736437060 (173h39m1.287907521s) [upload=152907]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 16 13:43:02.736444273 (173h39m1.287914735s) [upload=152907]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 16 13:43:05.031396865 (173h39m3.582867336s) [upload=152907]: Done uploading segments... Nov 16 13:43:05.031422193 (173h39m3.582892654s) [upload=152907]: Upload finished. err: <nil> Duration: 6.186332153s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 6.140144601s trace: https://jaeger.dev.storj.io/trace/3c77028f820d12f9 FAIL (2025-11-16 14:26:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, Piece ID: UKP3MHXBSALCWFIHCTSR5JFFXEWWAWGDS4JYUYS2JEREQQKTPJSA) context deadline exceeded; piecestore: (Node ID: 12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, Piece ID: DNT5QVAMRO5OMBNVTDRRUKACMM7M5472LD5E4ASKYIBKBNWYAVRA) context deadline exceeded; piecestore: (Node ID: 17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, Piece ID: DPLALMJC4JPC5D7BOK5SOSDFDJFDDPGSJWP4XDDQE22AMFNAKATQ) context deadline exceeded ======LOGS====== gment. Inline: false Nov 16 13:42:52.666975419 (173h38m51.21844588s) [upload=152907]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 16 13:42:52.666980539 (173h38m51.21845099s) [upload=152907]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 16 13:42:53.024947049 (173h38m51.5764175s) [seg_pos={0 0}, upload=152907]: Begin upload segment... Nov 16 13:42:53.025007074 (173h38m51.576477535s) [seg_pos={0 0}, upload=152907]: Done begin upload segment. Nov 16 13:42:53.025087707 (173h38m51.576558158s) [noise=true , address=77.90.31.14:4662 , piece_id=S5PPMCKIKSOCLUJ4CEH7ZDW3TVUZ7GTFFEIPCQK6FX25UIOUQHHQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.025098377 (173h38m51.576568828s) [noise=true , address=145.79.227.31:4797 , piece_id=UKP3MHXBSALCWFIHCTSR5JFFXEWWAWGDS4JYUYS2JEREQQKTPJSA, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.025961451 (173h38m51.577431912s) [noise=true , address=45.91.116.6:4974 , piece_id=ZTS3H7E6A3VJECLRRKRKSVX5PH3VSH7F2HAUPQSPH56NCI6RTFBQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.026101888 (173h38m51.577572339s) [noise=true , address=145.79.227.10:4402 , piece_id=DNT5QVAMRO5OMBNVTDRRUKACMM7M5472LD5E4ASKYIBKBNWYAVRA, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.026397721 (173h38m51.577868182s) [noise=true , address=145.79.227.11:4732 , piece_id=GR6SASHNUTHAEOUPK7DWKVPHRGMG72FQMF4RINJ5NTIQQGFCVZZQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.026543650 (173h38m51.578014111s) [noise=true , address=145.79.227.20:4437 , piece_id=G5P6OYXVAAQDZKTPPHDNZLMYTT6L2TFWKH7N4F7L6742BWHOQGBQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.026887915 (173h38m51.578358366s) [noise=true , address=45.91.116.14:4189 , piece_id=NUMS6Q3F2OUAQ2PSKE2CZK2WKRQLZUZPWVBA3UA6UNKDQACPYSJQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.027039624 (173h38m51.578510085s) [noise=true , address=145.79.227.27:4691 , piece_id=NSX5S2D7ZSA5I57B4EAQEQXEVUMPKNMRKRV3RHWGWJ2DN7SWUPGQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.027374482 (173h38m51.578844973s) [noise=true , address=145.79.227.31:4798 , piece_id=DPLALMJC4JPC5D7BOK5SOSDFDJFDDPGSJWP4XDDQE22AMFNAKATQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.027472337 (173h38m51.578942798s) [noise=true , address=45.91.116.7:4010 , piece_id=EO7IEOCWEJAMZZU4BLIB5ALFACWTWZE3QCXU3FUY4FKC4P444ORA, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.027920821 (173h38m51.579391283s) [noise=true , address=145.79.227.23:4547 , piece_id=P4WX2ZDHV2Z6BFMF6IACIYW4DXKYFVSULUTF7EK2EMRO5OA5X5RQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.028194272 (173h38m51.579664743s) [noise=true , address=145.79.227.32:4834 , piece_id=CBK2UQ252EPFQDTOYS2X5WAUTJN4L2DTHGIOJXFZXWBOLGPRETXQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.028430262 (173h38m51.579900723s) [noise=true , address=145.79.227.9:4366 , piece_id=YFWF2YL2YPYVMP5DHJKRUGVUUS2ALG4DRJ3HUZCDZM4N3N676FNQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.029687726 (173h38m51.581158177s) [noise=true , address=145.79.227.20:4438 , piece_id=6HY2HDNJAZ552ZKXFJ7VNTQNBAQU5ZQLNKAG2NZZWZQHN6EEDG6Q, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.031784128 (173h38m51.583254589s) [noise=true , address=45.91.116.5:4298 , piece_id=ARGPAZE6CTMM6UXSMPSVHDBEQWKVGKY7WXOVLPB7UH3JZYMDB7JQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.034100830 (173h38m51.585571291s) [noise=true , address=77.90.31.10:4518 , piece_id=MSKQ66KE4WANAE4AWOD2IKY3PQN2YGZ5N4W3NXGKXSOLE7B6H6WQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:54.850200282 (173h38m53.401670733s) [upload=152907]: Next returned nil segment Nov 16 13:42:54.850219118 (173h38m53.401689569s) [upload=152907]: Waiting for error group managing segments... Nov 16 13:42:55.154333032 (173h38m53.705803493s) [noise=true , address=45.91.116.7:4010 , piece_id=EO7IEOCWEJAMZZU4BLIB5ALFACWTWZE3QCXU3FUY4FKC4P444ORA, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:42:55.201689040 (173h38m53.753159501s) [noise=true , address=77.90.31.10:4518 , piece_id=MSKQ66KE4WANAE4AWOD2IKY3PQN2YGZ5N4W3NXGKXSOLE7B6H6WQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:42:55.232313345 (173h38m53.783783816s) [noise=true , address=45.91.116.5:4298 , piece_id=ARGPAZE6CTMM6UXSMPSVHDBEQWKVGKY7WXOVLPB7UH3JZYMDB7JQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:42:55.260281393 (173h38m53.811751854s) [noise=true , address=45.91.116.6:4974 , piece_id=ZTS3H7E6A3VJECLRRKRKSVX5PH3VSH7F2HAUPQSPH56NCI6RTFBQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:00.229127085 (173h38m58.780597546s) [noise=true , address=145.79.227.20:4438 , piece_id=6HY2HDNJAZ552ZKXFJ7VNTQNBAQU5ZQLNKAG2NZZWZQHN6EEDG6Q, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:00.358241119 (173h38m58.909711581s) [noise=true , address=145.79.227.32:4834 , piece_id=CBK2UQ252EPFQDTOYS2X5WAUTJN4L2DTHGIOJXFZXWBOLGPRETXQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:00.646958215 (173h38m59.198428676s) [noise=true , address=145.79.227.31:4798 , piece_id=DPLALMJC4JPC5D7BOK5SOSDFDJFDDPGSJWP4XDDQE22AMFNAKATQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:00.908824080 (173h38m59.460294541s) [noise=true , address=145.79.227.23:4547 , piece_id=P4WX2ZDHV2Z6BFMF6IACIYW4DXKYFVSULUTF7EK2EMRO5OA5X5RQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:01.155346527 (173h38m59.706816989s) [noise=true , address=45.91.116.14:4189 , piece_id=NUMS6Q3F2OUAQ2PSKE2CZK2WKRQLZUZPWVBA3UA6UNKDQACPYSJQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:01.839444794 (173h39m0.390915255s) [noise=true , address=145.79.227.11:4732 , piece_id=GR6SASHNUTHAEOUPK7DWKVPHRGMG72FQMF4RINJ5NTIQQGFCVZZQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:02.591249579 (173h39m1.14272004s) [noise=true , address=145.79.227.10:4402 , piece_id=DNT5QVAMRO5OMBNVTDRRUKACMM7M5472LD5E4ASKYIBKBNWYAVRA, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:02.616126740 (173h39m1.167597202s) [noise=true , address=145.79.227.31:4797 , piece_id=UKP3MHXBSALCWFIHCTSR5JFFXEWWAWGDS4JYUYS2JEREQQKTPJSA, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:02.734605482 (173h39m1.286075943s) [noise=true , address=145.79.227.27:4691 , piece_id=NSX5S2D7ZSA5I57B4EAQEQXEVUMPKNMRKRV3RHWGWJ2DN7SWUPGQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:02.734638004 (173h39m1.286108475s) [seg_pos={0 0}, upload=152907]: Segment reached optimal threshold of 13 pieces. Nov 16 13:43:02.735049748 (173h39m1.286520209s) [noise=true , address=145.79.227.9:4366 , piece_id=YFWF2YL2YPYVMP5DHJKRUGVUUS2ALG4DRJ3HUZCDZM4N3N676FNQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy): protocol: write tcp 188.166.225.170:60100->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:60100->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60100->145.79.227.9:4366: use of closed network connection Nov 16 13:43:02.735060949 (173h39m1.28653141s) [noise=true , address=145.79.227.9:4366 , piece_id=YFWF2YL2YPYVMP5DHJKRUGVUUS2ALG4DRJ3HUZCDZM4N3N676FNQ, seg_pos={0 0}, upload=152907]: Long tail cancellation due to optimal threshold Nov 16 13:43:02.735509193 (173h39m1.286979654s) [noise=true , address=77.90.31.14:4662 , piece_id=S5PPMCKIKSOCLUJ4CEH7ZDW3TVUZ7GTFFEIPCQK6FX25UIOUQHHQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): protocol: write tcp 188.166.225.170:50196->77.90.31.14:4662: use of closed network connection; write tcp 188.166.225.170:50196->77.90.31.14:4662: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50196->77.90.31.14:4662: use of closed network connection Nov 16 13:43:02.735518792 (173h39m1.286989253s) [noise=true , address=77.90.31.14:4662 , piece_id=S5PPMCKIKSOCLUJ4CEH7ZDW3TVUZ7GTFFEIPCQK6FX25UIOUQHHQ, seg_pos={0 0}, upload=152907]: Long tail cancellation due to optimal threshold Nov 16 13:43:02.736224335 (173h39m1.287694796s) [noise=true , address=145.79.227.20:4437 , piece_id=G5P6OYXVAAQDZKTPPHDNZLMYTT6L2TFWKH7N4F7L6742BWHOQGBQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV): protocol: read tcp 188.166.225.170:35300->145.79.227.20:4437: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 13:43:02.736239203 (173h39m1.287709654s) [noise=true , address=145.79.227.20:4437 , piece_id=G5P6OYXVAAQDZKTPPHDNZLMYTT6L2TFWKH7N4F7L6742BWHOQGBQ, seg_pos={0 0}, upload=152907]: Long tail cancellation due to optimal threshold Nov 16 13:43:02.736270833 (173h39m1.287741294s) [seg_pos={0 0}, upload=152907]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 16 13:43:02.736437060 (173h39m1.287907521s) [upload=152907]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 16 13:43:02.736444273 (173h39m1.287914735s) [upload=152907]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 16 13:43:05.031396865 (173h39m3.582867336s) [upload=152907]: Done uploading segments... Nov 16 13:43:05.031422193 (173h39m3.582892654s) [upload=152907]: Upload finished. err: <nil> Duration: 6.140178306s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 10 out of 60 is failed (threshold=6) [..XX....X......................X........X..X.....X..XX...X..] duration: 3.577892096s trace: https://jaeger.dev.storj.io/trace/4fb8f2cf6fee9d8c FAIL (2025-11-16 14:26:48 UTC) orbiter warning: context deadline exceeded Duration: 3.577910741s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 11 out of 60 is failed (threshold=6) [X..XX....X......................X........X..X.....X..XX...X.] duration: 3.467423002s trace: https://jaeger.dev.storj.io/trace/4fb8f2cf6fee9d8c FAIL (2025-11-16 14:25:52 UTC) orbiter warning: context deadline exceeded Duration: 3.467456154s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 6.085402311s trace: https://jaeger.dev.storj.io/trace/3c77028f820d12f9 FAIL (2025-11-16 14:25:40 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, Piece ID: UKP3MHXBSALCWFIHCTSR5JFFXEWWAWGDS4JYUYS2JEREQQKTPJSA) context deadline exceeded; piecestore: (Node ID: 12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, Piece ID: DNT5QVAMRO5OMBNVTDRRUKACMM7M5472LD5E4ASKYIBKBNWYAVRA) context deadline exceeded; piecestore: (Node ID: 17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, Piece ID: DPLALMJC4JPC5D7BOK5SOSDFDJFDDPGSJWP4XDDQE22AMFNAKATQ) context deadline exceeded ======LOGS====== gment. Inline: false Nov 16 13:42:52.666975419 (173h38m51.21844588s) [upload=152907]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 16 13:42:52.666980539 (173h38m51.21845099s) [upload=152907]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 16 13:42:53.024947049 (173h38m51.5764175s) [seg_pos={0 0}, upload=152907]: Begin upload segment... Nov 16 13:42:53.025007074 (173h38m51.576477535s) [seg_pos={0 0}, upload=152907]: Done begin upload segment. Nov 16 13:42:53.025087707 (173h38m51.576558158s) [noise=true , address=77.90.31.14:4662 , piece_id=S5PPMCKIKSOCLUJ4CEH7ZDW3TVUZ7GTFFEIPCQK6FX25UIOUQHHQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.025098377 (173h38m51.576568828s) [noise=true , address=145.79.227.31:4797 , piece_id=UKP3MHXBSALCWFIHCTSR5JFFXEWWAWGDS4JYUYS2JEREQQKTPJSA, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.025961451 (173h38m51.577431912s) [noise=true , address=45.91.116.6:4974 , piece_id=ZTS3H7E6A3VJECLRRKRKSVX5PH3VSH7F2HAUPQSPH56NCI6RTFBQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.026101888 (173h38m51.577572339s) [noise=true , address=145.79.227.10:4402 , piece_id=DNT5QVAMRO5OMBNVTDRRUKACMM7M5472LD5E4ASKYIBKBNWYAVRA, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.026397721 (173h38m51.577868182s) [noise=true , address=145.79.227.11:4732 , piece_id=GR6SASHNUTHAEOUPK7DWKVPHRGMG72FQMF4RINJ5NTIQQGFCVZZQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.026543650 (173h38m51.578014111s) [noise=true , address=145.79.227.20:4437 , piece_id=G5P6OYXVAAQDZKTPPHDNZLMYTT6L2TFWKH7N4F7L6742BWHOQGBQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.026887915 (173h38m51.578358366s) [noise=true , address=45.91.116.14:4189 , piece_id=NUMS6Q3F2OUAQ2PSKE2CZK2WKRQLZUZPWVBA3UA6UNKDQACPYSJQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.027039624 (173h38m51.578510085s) [noise=true , address=145.79.227.27:4691 , piece_id=NSX5S2D7ZSA5I57B4EAQEQXEVUMPKNMRKRV3RHWGWJ2DN7SWUPGQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.027374482 (173h38m51.578844973s) [noise=true , address=145.79.227.31:4798 , piece_id=DPLALMJC4JPC5D7BOK5SOSDFDJFDDPGSJWP4XDDQE22AMFNAKATQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.027472337 (173h38m51.578942798s) [noise=true , address=45.91.116.7:4010 , piece_id=EO7IEOCWEJAMZZU4BLIB5ALFACWTWZE3QCXU3FUY4FKC4P444ORA, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.027920821 (173h38m51.579391283s) [noise=true , address=145.79.227.23:4547 , piece_id=P4WX2ZDHV2Z6BFMF6IACIYW4DXKYFVSULUTF7EK2EMRO5OA5X5RQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.028194272 (173h38m51.579664743s) [noise=true , address=145.79.227.32:4834 , piece_id=CBK2UQ252EPFQDTOYS2X5WAUTJN4L2DTHGIOJXFZXWBOLGPRETXQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.028430262 (173h38m51.579900723s) [noise=true , address=145.79.227.9:4366 , piece_id=YFWF2YL2YPYVMP5DHJKRUGVUUS2ALG4DRJ3HUZCDZM4N3N676FNQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.029687726 (173h38m51.581158177s) [noise=true , address=145.79.227.20:4438 , piece_id=6HY2HDNJAZ552ZKXFJ7VNTQNBAQU5ZQLNKAG2NZZWZQHN6EEDG6Q, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.031784128 (173h38m51.583254589s) [noise=true , address=45.91.116.5:4298 , piece_id=ARGPAZE6CTMM6UXSMPSVHDBEQWKVGKY7WXOVLPB7UH3JZYMDB7JQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.034100830 (173h38m51.585571291s) [noise=true , address=77.90.31.10:4518 , piece_id=MSKQ66KE4WANAE4AWOD2IKY3PQN2YGZ5N4W3NXGKXSOLE7B6H6WQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:54.850200282 (173h38m53.401670733s) [upload=152907]: Next returned nil segment Nov 16 13:42:54.850219118 (173h38m53.401689569s) [upload=152907]: Waiting for error group managing segments... Nov 16 13:42:55.154333032 (173h38m53.705803493s) [noise=true , address=45.91.116.7:4010 , piece_id=EO7IEOCWEJAMZZU4BLIB5ALFACWTWZE3QCXU3FUY4FKC4P444ORA, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:42:55.201689040 (173h38m53.753159501s) [noise=true , address=77.90.31.10:4518 , piece_id=MSKQ66KE4WANAE4AWOD2IKY3PQN2YGZ5N4W3NXGKXSOLE7B6H6WQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:42:55.232313345 (173h38m53.783783816s) [noise=true , address=45.91.116.5:4298 , piece_id=ARGPAZE6CTMM6UXSMPSVHDBEQWKVGKY7WXOVLPB7UH3JZYMDB7JQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:42:55.260281393 (173h38m53.811751854s) [noise=true , address=45.91.116.6:4974 , piece_id=ZTS3H7E6A3VJECLRRKRKSVX5PH3VSH7F2HAUPQSPH56NCI6RTFBQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:00.229127085 (173h38m58.780597546s) [noise=true , address=145.79.227.20:4438 , piece_id=6HY2HDNJAZ552ZKXFJ7VNTQNBAQU5ZQLNKAG2NZZWZQHN6EEDG6Q, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:00.358241119 (173h38m58.909711581s) [noise=true , address=145.79.227.32:4834 , piece_id=CBK2UQ252EPFQDTOYS2X5WAUTJN4L2DTHGIOJXFZXWBOLGPRETXQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:00.646958215 (173h38m59.198428676s) [noise=true , address=145.79.227.31:4798 , piece_id=DPLALMJC4JPC5D7BOK5SOSDFDJFDDPGSJWP4XDDQE22AMFNAKATQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:00.908824080 (173h38m59.460294541s) [noise=true , address=145.79.227.23:4547 , piece_id=P4WX2ZDHV2Z6BFMF6IACIYW4DXKYFVSULUTF7EK2EMRO5OA5X5RQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:01.155346527 (173h38m59.706816989s) [noise=true , address=45.91.116.14:4189 , piece_id=NUMS6Q3F2OUAQ2PSKE2CZK2WKRQLZUZPWVBA3UA6UNKDQACPYSJQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:01.839444794 (173h39m0.390915255s) [noise=true , address=145.79.227.11:4732 , piece_id=GR6SASHNUTHAEOUPK7DWKVPHRGMG72FQMF4RINJ5NTIQQGFCVZZQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:02.591249579 (173h39m1.14272004s) [noise=true , address=145.79.227.10:4402 , piece_id=DNT5QVAMRO5OMBNVTDRRUKACMM7M5472LD5E4ASKYIBKBNWYAVRA, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:02.616126740 (173h39m1.167597202s) [noise=true , address=145.79.227.31:4797 , piece_id=UKP3MHXBSALCWFIHCTSR5JFFXEWWAWGDS4JYUYS2JEREQQKTPJSA, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:02.734605482 (173h39m1.286075943s) [noise=true , address=145.79.227.27:4691 , piece_id=NSX5S2D7ZSA5I57B4EAQEQXEVUMPKNMRKRV3RHWGWJ2DN7SWUPGQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:02.734638004 (173h39m1.286108475s) [seg_pos={0 0}, upload=152907]: Segment reached optimal threshold of 13 pieces. Nov 16 13:43:02.735049748 (173h39m1.286520209s) [noise=true , address=145.79.227.9:4366 , piece_id=YFWF2YL2YPYVMP5DHJKRUGVUUS2ALG4DRJ3HUZCDZM4N3N676FNQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy): protocol: write tcp 188.166.225.170:60100->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:60100->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60100->145.79.227.9:4366: use of closed network connection Nov 16 13:43:02.735060949 (173h39m1.28653141s) [noise=true , address=145.79.227.9:4366 , piece_id=YFWF2YL2YPYVMP5DHJKRUGVUUS2ALG4DRJ3HUZCDZM4N3N676FNQ, seg_pos={0 0}, upload=152907]: Long tail cancellation due to optimal threshold Nov 16 13:43:02.735509193 (173h39m1.286979654s) [noise=true , address=77.90.31.14:4662 , piece_id=S5PPMCKIKSOCLUJ4CEH7ZDW3TVUZ7GTFFEIPCQK6FX25UIOUQHHQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): protocol: write tcp 188.166.225.170:50196->77.90.31.14:4662: use of closed network connection; write tcp 188.166.225.170:50196->77.90.31.14:4662: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50196->77.90.31.14:4662: use of closed network connection Nov 16 13:43:02.735518792 (173h39m1.286989253s) [noise=true , address=77.90.31.14:4662 , piece_id=S5PPMCKIKSOCLUJ4CEH7ZDW3TVUZ7GTFFEIPCQK6FX25UIOUQHHQ, seg_pos={0 0}, upload=152907]: Long tail cancellation due to optimal threshold Nov 16 13:43:02.736224335 (173h39m1.287694796s) [noise=true , address=145.79.227.20:4437 , piece_id=G5P6OYXVAAQDZKTPPHDNZLMYTT6L2TFWKH7N4F7L6742BWHOQGBQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV): protocol: read tcp 188.166.225.170:35300->145.79.227.20:4437: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 13:43:02.736239203 (173h39m1.287709654s) [noise=true , address=145.79.227.20:4437 , piece_id=G5P6OYXVAAQDZKTPPHDNZLMYTT6L2TFWKH7N4F7L6742BWHOQGBQ, seg_pos={0 0}, upload=152907]: Long tail cancellation due to optimal threshold Nov 16 13:43:02.736270833 (173h39m1.287741294s) [seg_pos={0 0}, upload=152907]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 16 13:43:02.736437060 (173h39m1.287907521s) [upload=152907]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 16 13:43:02.736444273 (173h39m1.287914735s) [upload=152907]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 16 13:43:05.031396865 (173h39m3.582867336s) [upload=152907]: Done uploading segments... Nov 16 13:43:05.031422193 (173h39m3.582892654s) [upload=152907]: Upload finished. err: <nil> Duration: 6.085429852s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 11 out of 60 is failed (threshold=6) [.X..XX....X......................X........X..X.....X..XX...X] duration: 5.001316764s trace: https://jaeger.dev.storj.io/trace/4fb8f2cf6fee9d8c FAIL (2025-11-16 14:24:51 UTC) orbiter warning: context deadline exceeded Duration: 5.001344297s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 7.170102965s trace: https://jaeger.dev.storj.io/trace/3c77028f820d12f9 FAIL (2025-11-16 14:24:30 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, Piece ID: UKP3MHXBSALCWFIHCTSR5JFFXEWWAWGDS4JYUYS2JEREQQKTPJSA) context deadline exceeded; piecestore: (Node ID: 12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, Piece ID: DNT5QVAMRO5OMBNVTDRRUKACMM7M5472LD5E4ASKYIBKBNWYAVRA) context deadline exceeded; piecestore: (Node ID: 17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, Piece ID: DPLALMJC4JPC5D7BOK5SOSDFDJFDDPGSJWP4XDDQE22AMFNAKATQ) context deadline exceeded ======LOGS====== gment. Inline: false Nov 16 13:42:52.666975419 (173h38m51.21844588s) [upload=152907]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 16 13:42:52.666980539 (173h38m51.21845099s) [upload=152907]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 16 13:42:53.024947049 (173h38m51.5764175s) [seg_pos={0 0}, upload=152907]: Begin upload segment... Nov 16 13:42:53.025007074 (173h38m51.576477535s) [seg_pos={0 0}, upload=152907]: Done begin upload segment. Nov 16 13:42:53.025087707 (173h38m51.576558158s) [noise=true , address=77.90.31.14:4662 , piece_id=S5PPMCKIKSOCLUJ4CEH7ZDW3TVUZ7GTFFEIPCQK6FX25UIOUQHHQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.025098377 (173h38m51.576568828s) [noise=true , address=145.79.227.31:4797 , piece_id=UKP3MHXBSALCWFIHCTSR5JFFXEWWAWGDS4JYUYS2JEREQQKTPJSA, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.025961451 (173h38m51.577431912s) [noise=true , address=45.91.116.6:4974 , piece_id=ZTS3H7E6A3VJECLRRKRKSVX5PH3VSH7F2HAUPQSPH56NCI6RTFBQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.026101888 (173h38m51.577572339s) [noise=true , address=145.79.227.10:4402 , piece_id=DNT5QVAMRO5OMBNVTDRRUKACMM7M5472LD5E4ASKYIBKBNWYAVRA, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.026397721 (173h38m51.577868182s) [noise=true , address=145.79.227.11:4732 , piece_id=GR6SASHNUTHAEOUPK7DWKVPHRGMG72FQMF4RINJ5NTIQQGFCVZZQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.026543650 (173h38m51.578014111s) [noise=true , address=145.79.227.20:4437 , piece_id=G5P6OYXVAAQDZKTPPHDNZLMYTT6L2TFWKH7N4F7L6742BWHOQGBQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.026887915 (173h38m51.578358366s) [noise=true , address=45.91.116.14:4189 , piece_id=NUMS6Q3F2OUAQ2PSKE2CZK2WKRQLZUZPWVBA3UA6UNKDQACPYSJQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.027039624 (173h38m51.578510085s) [noise=true , address=145.79.227.27:4691 , piece_id=NSX5S2D7ZSA5I57B4EAQEQXEVUMPKNMRKRV3RHWGWJ2DN7SWUPGQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.027374482 (173h38m51.578844973s) [noise=true , address=145.79.227.31:4798 , piece_id=DPLALMJC4JPC5D7BOK5SOSDFDJFDDPGSJWP4XDDQE22AMFNAKATQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.027472337 (173h38m51.578942798s) [noise=true , address=45.91.116.7:4010 , piece_id=EO7IEOCWEJAMZZU4BLIB5ALFACWTWZE3QCXU3FUY4FKC4P444ORA, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.027920821 (173h38m51.579391283s) [noise=true , address=145.79.227.23:4547 , piece_id=P4WX2ZDHV2Z6BFMF6IACIYW4DXKYFVSULUTF7EK2EMRO5OA5X5RQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.028194272 (173h38m51.579664743s) [noise=true , address=145.79.227.32:4834 , piece_id=CBK2UQ252EPFQDTOYS2X5WAUTJN4L2DTHGIOJXFZXWBOLGPRETXQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.028430262 (173h38m51.579900723s) [noise=true , address=145.79.227.9:4366 , piece_id=YFWF2YL2YPYVMP5DHJKRUGVUUS2ALG4DRJ3HUZCDZM4N3N676FNQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.029687726 (173h38m51.581158177s) [noise=true , address=145.79.227.20:4438 , piece_id=6HY2HDNJAZ552ZKXFJ7VNTQNBAQU5ZQLNKAG2NZZWZQHN6EEDG6Q, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.031784128 (173h38m51.583254589s) [noise=true , address=45.91.116.5:4298 , piece_id=ARGPAZE6CTMM6UXSMPSVHDBEQWKVGKY7WXOVLPB7UH3JZYMDB7JQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.034100830 (173h38m51.585571291s) [noise=true , address=77.90.31.10:4518 , piece_id=MSKQ66KE4WANAE4AWOD2IKY3PQN2YGZ5N4W3NXGKXSOLE7B6H6WQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:54.850200282 (173h38m53.401670733s) [upload=152907]: Next returned nil segment Nov 16 13:42:54.850219118 (173h38m53.401689569s) [upload=152907]: Waiting for error group managing segments... Nov 16 13:42:55.154333032 (173h38m53.705803493s) [noise=true , address=45.91.116.7:4010 , piece_id=EO7IEOCWEJAMZZU4BLIB5ALFACWTWZE3QCXU3FUY4FKC4P444ORA, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:42:55.201689040 (173h38m53.753159501s) [noise=true , address=77.90.31.10:4518 , piece_id=MSKQ66KE4WANAE4AWOD2IKY3PQN2YGZ5N4W3NXGKXSOLE7B6H6WQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:42:55.232313345 (173h38m53.783783816s) [noise=true , address=45.91.116.5:4298 , piece_id=ARGPAZE6CTMM6UXSMPSVHDBEQWKVGKY7WXOVLPB7UH3JZYMDB7JQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:42:55.260281393 (173h38m53.811751854s) [noise=true , address=45.91.116.6:4974 , piece_id=ZTS3H7E6A3VJECLRRKRKSVX5PH3VSH7F2HAUPQSPH56NCI6RTFBQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:00.229127085 (173h38m58.780597546s) [noise=true , address=145.79.227.20:4438 , piece_id=6HY2HDNJAZ552ZKXFJ7VNTQNBAQU5ZQLNKAG2NZZWZQHN6EEDG6Q, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:00.358241119 (173h38m58.909711581s) [noise=true , address=145.79.227.32:4834 , piece_id=CBK2UQ252EPFQDTOYS2X5WAUTJN4L2DTHGIOJXFZXWBOLGPRETXQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:00.646958215 (173h38m59.198428676s) [noise=true , address=145.79.227.31:4798 , piece_id=DPLALMJC4JPC5D7BOK5SOSDFDJFDDPGSJWP4XDDQE22AMFNAKATQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:00.908824080 (173h38m59.460294541s) [noise=true , address=145.79.227.23:4547 , piece_id=P4WX2ZDHV2Z6BFMF6IACIYW4DXKYFVSULUTF7EK2EMRO5OA5X5RQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:01.155346527 (173h38m59.706816989s) [noise=true , address=45.91.116.14:4189 , piece_id=NUMS6Q3F2OUAQ2PSKE2CZK2WKRQLZUZPWVBA3UA6UNKDQACPYSJQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:01.839444794 (173h39m0.390915255s) [noise=true , address=145.79.227.11:4732 , piece_id=GR6SASHNUTHAEOUPK7DWKVPHRGMG72FQMF4RINJ5NTIQQGFCVZZQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:02.591249579 (173h39m1.14272004s) [noise=true , address=145.79.227.10:4402 , piece_id=DNT5QVAMRO5OMBNVTDRRUKACMM7M5472LD5E4ASKYIBKBNWYAVRA, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:02.616126740 (173h39m1.167597202s) [noise=true , address=145.79.227.31:4797 , piece_id=UKP3MHXBSALCWFIHCTSR5JFFXEWWAWGDS4JYUYS2JEREQQKTPJSA, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:02.734605482 (173h39m1.286075943s) [noise=true , address=145.79.227.27:4691 , piece_id=NSX5S2D7ZSA5I57B4EAQEQXEVUMPKNMRKRV3RHWGWJ2DN7SWUPGQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:02.734638004 (173h39m1.286108475s) [seg_pos={0 0}, upload=152907]: Segment reached optimal threshold of 13 pieces. Nov 16 13:43:02.735049748 (173h39m1.286520209s) [noise=true , address=145.79.227.9:4366 , piece_id=YFWF2YL2YPYVMP5DHJKRUGVUUS2ALG4DRJ3HUZCDZM4N3N676FNQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy): protocol: write tcp 188.166.225.170:60100->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:60100->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60100->145.79.227.9:4366: use of closed network connection Nov 16 13:43:02.735060949 (173h39m1.28653141s) [noise=true , address=145.79.227.9:4366 , piece_id=YFWF2YL2YPYVMP5DHJKRUGVUUS2ALG4DRJ3HUZCDZM4N3N676FNQ, seg_pos={0 0}, upload=152907]: Long tail cancellation due to optimal threshold Nov 16 13:43:02.735509193 (173h39m1.286979654s) [noise=true , address=77.90.31.14:4662 , piece_id=S5PPMCKIKSOCLUJ4CEH7ZDW3TVUZ7GTFFEIPCQK6FX25UIOUQHHQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): protocol: write tcp 188.166.225.170:50196->77.90.31.14:4662: use of closed network connection; write tcp 188.166.225.170:50196->77.90.31.14:4662: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50196->77.90.31.14:4662: use of closed network connection Nov 16 13:43:02.735518792 (173h39m1.286989253s) [noise=true , address=77.90.31.14:4662 , piece_id=S5PPMCKIKSOCLUJ4CEH7ZDW3TVUZ7GTFFEIPCQK6FX25UIOUQHHQ, seg_pos={0 0}, upload=152907]: Long tail cancellation due to optimal threshold Nov 16 13:43:02.736224335 (173h39m1.287694796s) [noise=true , address=145.79.227.20:4437 , piece_id=G5P6OYXVAAQDZKTPPHDNZLMYTT6L2TFWKH7N4F7L6742BWHOQGBQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV): protocol: read tcp 188.166.225.170:35300->145.79.227.20:4437: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 13:43:02.736239203 (173h39m1.287709654s) [noise=true , address=145.79.227.20:4437 , piece_id=G5P6OYXVAAQDZKTPPHDNZLMYTT6L2TFWKH7N4F7L6742BWHOQGBQ, seg_pos={0 0}, upload=152907]: Long tail cancellation due to optimal threshold Nov 16 13:43:02.736270833 (173h39m1.287741294s) [seg_pos={0 0}, upload=152907]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 16 13:43:02.736437060 (173h39m1.287907521s) [upload=152907]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 16 13:43:02.736444273 (173h39m1.287914735s) [upload=152907]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 16 13:43:05.031396865 (173h39m3.582867336s) [upload=152907]: Done uploading segments... Nov 16 13:43:05.031422193 (173h39m3.582892654s) [upload=152907]: Upload finished. err: <nil> Duration: 7.17013175s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 10 out of 60 is failed (threshold=6) [..X..XX....X......................X........X..X.....X..XX...] duration: 4.855948093s trace: https://jaeger.dev.storj.io/trace/16dd114e41a655be FAIL (2025-11-16 14:24:12 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.85597325s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 6.41304759s trace: https://jaeger.dev.storj.io/trace/3c77028f820d12f9 FAIL (2025-11-16 14:23:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, Piece ID: UKP3MHXBSALCWFIHCTSR5JFFXEWWAWGDS4JYUYS2JEREQQKTPJSA) context deadline exceeded; piecestore: (Node ID: 12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, Piece ID: DNT5QVAMRO5OMBNVTDRRUKACMM7M5472LD5E4ASKYIBKBNWYAVRA) context deadline exceeded; piecestore: (Node ID: 17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, Piece ID: DPLALMJC4JPC5D7BOK5SOSDFDJFDDPGSJWP4XDDQE22AMFNAKATQ) context deadline exceeded ======LOGS====== gment. Inline: false Nov 16 13:42:52.666975419 (173h38m51.21844588s) [upload=152907]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 16 13:42:52.666980539 (173h38m51.21845099s) [upload=152907]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 16 13:42:53.024947049 (173h38m51.5764175s) [seg_pos={0 0}, upload=152907]: Begin upload segment... Nov 16 13:42:53.025007074 (173h38m51.576477535s) [seg_pos={0 0}, upload=152907]: Done begin upload segment. Nov 16 13:42:53.025087707 (173h38m51.576558158s) [noise=true , address=77.90.31.14:4662 , piece_id=S5PPMCKIKSOCLUJ4CEH7ZDW3TVUZ7GTFFEIPCQK6FX25UIOUQHHQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.025098377 (173h38m51.576568828s) [noise=true , address=145.79.227.31:4797 , piece_id=UKP3MHXBSALCWFIHCTSR5JFFXEWWAWGDS4JYUYS2JEREQQKTPJSA, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.025961451 (173h38m51.577431912s) [noise=true , address=45.91.116.6:4974 , piece_id=ZTS3H7E6A3VJECLRRKRKSVX5PH3VSH7F2HAUPQSPH56NCI6RTFBQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.026101888 (173h38m51.577572339s) [noise=true , address=145.79.227.10:4402 , piece_id=DNT5QVAMRO5OMBNVTDRRUKACMM7M5472LD5E4ASKYIBKBNWYAVRA, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.026397721 (173h38m51.577868182s) [noise=true , address=145.79.227.11:4732 , piece_id=GR6SASHNUTHAEOUPK7DWKVPHRGMG72FQMF4RINJ5NTIQQGFCVZZQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.026543650 (173h38m51.578014111s) [noise=true , address=145.79.227.20:4437 , piece_id=G5P6OYXVAAQDZKTPPHDNZLMYTT6L2TFWKH7N4F7L6742BWHOQGBQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.026887915 (173h38m51.578358366s) [noise=true , address=45.91.116.14:4189 , piece_id=NUMS6Q3F2OUAQ2PSKE2CZK2WKRQLZUZPWVBA3UA6UNKDQACPYSJQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.027039624 (173h38m51.578510085s) [noise=true , address=145.79.227.27:4691 , piece_id=NSX5S2D7ZSA5I57B4EAQEQXEVUMPKNMRKRV3RHWGWJ2DN7SWUPGQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.027374482 (173h38m51.578844973s) [noise=true , address=145.79.227.31:4798 , piece_id=DPLALMJC4JPC5D7BOK5SOSDFDJFDDPGSJWP4XDDQE22AMFNAKATQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.027472337 (173h38m51.578942798s) [noise=true , address=45.91.116.7:4010 , piece_id=EO7IEOCWEJAMZZU4BLIB5ALFACWTWZE3QCXU3FUY4FKC4P444ORA, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.027920821 (173h38m51.579391283s) [noise=true , address=145.79.227.23:4547 , piece_id=P4WX2ZDHV2Z6BFMF6IACIYW4DXKYFVSULUTF7EK2EMRO5OA5X5RQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.028194272 (173h38m51.579664743s) [noise=true , address=145.79.227.32:4834 , piece_id=CBK2UQ252EPFQDTOYS2X5WAUTJN4L2DTHGIOJXFZXWBOLGPRETXQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.028430262 (173h38m51.579900723s) [noise=true , address=145.79.227.9:4366 , piece_id=YFWF2YL2YPYVMP5DHJKRUGVUUS2ALG4DRJ3HUZCDZM4N3N676FNQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.029687726 (173h38m51.581158177s) [noise=true , address=145.79.227.20:4438 , piece_id=6HY2HDNJAZ552ZKXFJ7VNTQNBAQU5ZQLNKAG2NZZWZQHN6EEDG6Q, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.031784128 (173h38m51.583254589s) [noise=true , address=45.91.116.5:4298 , piece_id=ARGPAZE6CTMM6UXSMPSVHDBEQWKVGKY7WXOVLPB7UH3JZYMDB7JQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.034100830 (173h38m51.585571291s) [noise=true , address=77.90.31.10:4518 , piece_id=MSKQ66KE4WANAE4AWOD2IKY3PQN2YGZ5N4W3NXGKXSOLE7B6H6WQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:54.850200282 (173h38m53.401670733s) [upload=152907]: Next returned nil segment Nov 16 13:42:54.850219118 (173h38m53.401689569s) [upload=152907]: Waiting for error group managing segments... Nov 16 13:42:55.154333032 (173h38m53.705803493s) [noise=true , address=45.91.116.7:4010 , piece_id=EO7IEOCWEJAMZZU4BLIB5ALFACWTWZE3QCXU3FUY4FKC4P444ORA, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:42:55.201689040 (173h38m53.753159501s) [noise=true , address=77.90.31.10:4518 , piece_id=MSKQ66KE4WANAE4AWOD2IKY3PQN2YGZ5N4W3NXGKXSOLE7B6H6WQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:42:55.232313345 (173h38m53.783783816s) [noise=true , address=45.91.116.5:4298 , piece_id=ARGPAZE6CTMM6UXSMPSVHDBEQWKVGKY7WXOVLPB7UH3JZYMDB7JQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:42:55.260281393 (173h38m53.811751854s) [noise=true , address=45.91.116.6:4974 , piece_id=ZTS3H7E6A3VJECLRRKRKSVX5PH3VSH7F2HAUPQSPH56NCI6RTFBQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:00.229127085 (173h38m58.780597546s) [noise=true , address=145.79.227.20:4438 , piece_id=6HY2HDNJAZ552ZKXFJ7VNTQNBAQU5ZQLNKAG2NZZWZQHN6EEDG6Q, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:00.358241119 (173h38m58.909711581s) [noise=true , address=145.79.227.32:4834 , piece_id=CBK2UQ252EPFQDTOYS2X5WAUTJN4L2DTHGIOJXFZXWBOLGPRETXQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:00.646958215 (173h38m59.198428676s) [noise=true , address=145.79.227.31:4798 , piece_id=DPLALMJC4JPC5D7BOK5SOSDFDJFDDPGSJWP4XDDQE22AMFNAKATQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:00.908824080 (173h38m59.460294541s) [noise=true , address=145.79.227.23:4547 , piece_id=P4WX2ZDHV2Z6BFMF6IACIYW4DXKYFVSULUTF7EK2EMRO5OA5X5RQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:01.155346527 (173h38m59.706816989s) [noise=true , address=45.91.116.14:4189 , piece_id=NUMS6Q3F2OUAQ2PSKE2CZK2WKRQLZUZPWVBA3UA6UNKDQACPYSJQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:01.839444794 (173h39m0.390915255s) [noise=true , address=145.79.227.11:4732 , piece_id=GR6SASHNUTHAEOUPK7DWKVPHRGMG72FQMF4RINJ5NTIQQGFCVZZQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:02.591249579 (173h39m1.14272004s) [noise=true , address=145.79.227.10:4402 , piece_id=DNT5QVAMRO5OMBNVTDRRUKACMM7M5472LD5E4ASKYIBKBNWYAVRA, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:02.616126740 (173h39m1.167597202s) [noise=true , address=145.79.227.31:4797 , piece_id=UKP3MHXBSALCWFIHCTSR5JFFXEWWAWGDS4JYUYS2JEREQQKTPJSA, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:02.734605482 (173h39m1.286075943s) [noise=true , address=145.79.227.27:4691 , piece_id=NSX5S2D7ZSA5I57B4EAQEQXEVUMPKNMRKRV3RHWGWJ2DN7SWUPGQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:02.734638004 (173h39m1.286108475s) [seg_pos={0 0}, upload=152907]: Segment reached optimal threshold of 13 pieces. Nov 16 13:43:02.735049748 (173h39m1.286520209s) [noise=true , address=145.79.227.9:4366 , piece_id=YFWF2YL2YPYVMP5DHJKRUGVUUS2ALG4DRJ3HUZCDZM4N3N676FNQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy): protocol: write tcp 188.166.225.170:60100->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:60100->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60100->145.79.227.9:4366: use of closed network connection Nov 16 13:43:02.735060949 (173h39m1.28653141s) [noise=true , address=145.79.227.9:4366 , piece_id=YFWF2YL2YPYVMP5DHJKRUGVUUS2ALG4DRJ3HUZCDZM4N3N676FNQ, seg_pos={0 0}, upload=152907]: Long tail cancellation due to optimal threshold Nov 16 13:43:02.735509193 (173h39m1.286979654s) [noise=true , address=77.90.31.14:4662 , piece_id=S5PPMCKIKSOCLUJ4CEH7ZDW3TVUZ7GTFFEIPCQK6FX25UIOUQHHQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): protocol: write tcp 188.166.225.170:50196->77.90.31.14:4662: use of closed network connection; write tcp 188.166.225.170:50196->77.90.31.14:4662: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50196->77.90.31.14:4662: use of closed network connection Nov 16 13:43:02.735518792 (173h39m1.286989253s) [noise=true , address=77.90.31.14:4662 , piece_id=S5PPMCKIKSOCLUJ4CEH7ZDW3TVUZ7GTFFEIPCQK6FX25UIOUQHHQ, seg_pos={0 0}, upload=152907]: Long tail cancellation due to optimal threshold Nov 16 13:43:02.736224335 (173h39m1.287694796s) [noise=true , address=145.79.227.20:4437 , piece_id=G5P6OYXVAAQDZKTPPHDNZLMYTT6L2TFWKH7N4F7L6742BWHOQGBQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV): protocol: read tcp 188.166.225.170:35300->145.79.227.20:4437: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 13:43:02.736239203 (173h39m1.287709654s) [noise=true , address=145.79.227.20:4437 , piece_id=G5P6OYXVAAQDZKTPPHDNZLMYTT6L2TFWKH7N4F7L6742BWHOQGBQ, seg_pos={0 0}, upload=152907]: Long tail cancellation due to optimal threshold Nov 16 13:43:02.736270833 (173h39m1.287741294s) [seg_pos={0 0}, upload=152907]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 16 13:43:02.736437060 (173h39m1.287907521s) [upload=152907]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 16 13:43:02.736444273 (173h39m1.287914735s) [upload=152907]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 16 13:43:05.031396865 (173h39m3.582867336s) [upload=152907]: Done uploading segments... Nov 16 13:43:05.031422193 (173h39m3.582892654s) [upload=152907]: Upload finished. err: <nil> Duration: 6.41307955s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 10 out of 60 is failed (threshold=6) [...X..XX....X......................X........X..X.....X..XX..] duration: 3.408492655s trace: https://jaeger.dev.storj.io/trace/16dd114e41a655be FAIL (2025-11-16 14:23:13 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.408523343s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 6.020889341s trace: https://jaeger.dev.storj.io/trace/3c77028f820d12f9 FAIL (2025-11-16 14:22:38 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, Piece ID: UKP3MHXBSALCWFIHCTSR5JFFXEWWAWGDS4JYUYS2JEREQQKTPJSA) context deadline exceeded; piecestore: (Node ID: 12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, Piece ID: DNT5QVAMRO5OMBNVTDRRUKACMM7M5472LD5E4ASKYIBKBNWYAVRA) context deadline exceeded; piecestore: (Node ID: 17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, Piece ID: DPLALMJC4JPC5D7BOK5SOSDFDJFDDPGSJWP4XDDQE22AMFNAKATQ) context deadline exceeded ======LOGS====== gment. Inline: false Nov 16 13:42:52.666975419 (173h38m51.21844588s) [upload=152907]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 16 13:42:52.666980539 (173h38m51.21845099s) [upload=152907]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 16 13:42:53.024947049 (173h38m51.5764175s) [seg_pos={0 0}, upload=152907]: Begin upload segment... Nov 16 13:42:53.025007074 (173h38m51.576477535s) [seg_pos={0 0}, upload=152907]: Done begin upload segment. Nov 16 13:42:53.025087707 (173h38m51.576558158s) [noise=true , address=77.90.31.14:4662 , piece_id=S5PPMCKIKSOCLUJ4CEH7ZDW3TVUZ7GTFFEIPCQK6FX25UIOUQHHQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.025098377 (173h38m51.576568828s) [noise=true , address=145.79.227.31:4797 , piece_id=UKP3MHXBSALCWFIHCTSR5JFFXEWWAWGDS4JYUYS2JEREQQKTPJSA, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.025961451 (173h38m51.577431912s) [noise=true , address=45.91.116.6:4974 , piece_id=ZTS3H7E6A3VJECLRRKRKSVX5PH3VSH7F2HAUPQSPH56NCI6RTFBQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.026101888 (173h38m51.577572339s) [noise=true , address=145.79.227.10:4402 , piece_id=DNT5QVAMRO5OMBNVTDRRUKACMM7M5472LD5E4ASKYIBKBNWYAVRA, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.026397721 (173h38m51.577868182s) [noise=true , address=145.79.227.11:4732 , piece_id=GR6SASHNUTHAEOUPK7DWKVPHRGMG72FQMF4RINJ5NTIQQGFCVZZQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.026543650 (173h38m51.578014111s) [noise=true , address=145.79.227.20:4437 , piece_id=G5P6OYXVAAQDZKTPPHDNZLMYTT6L2TFWKH7N4F7L6742BWHOQGBQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.026887915 (173h38m51.578358366s) [noise=true , address=45.91.116.14:4189 , piece_id=NUMS6Q3F2OUAQ2PSKE2CZK2WKRQLZUZPWVBA3UA6UNKDQACPYSJQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.027039624 (173h38m51.578510085s) [noise=true , address=145.79.227.27:4691 , piece_id=NSX5S2D7ZSA5I57B4EAQEQXEVUMPKNMRKRV3RHWGWJ2DN7SWUPGQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.027374482 (173h38m51.578844973s) [noise=true , address=145.79.227.31:4798 , piece_id=DPLALMJC4JPC5D7BOK5SOSDFDJFDDPGSJWP4XDDQE22AMFNAKATQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.027472337 (173h38m51.578942798s) [noise=true , address=45.91.116.7:4010 , piece_id=EO7IEOCWEJAMZZU4BLIB5ALFACWTWZE3QCXU3FUY4FKC4P444ORA, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.027920821 (173h38m51.579391283s) [noise=true , address=145.79.227.23:4547 , piece_id=P4WX2ZDHV2Z6BFMF6IACIYW4DXKYFVSULUTF7EK2EMRO5OA5X5RQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.028194272 (173h38m51.579664743s) [noise=true , address=145.79.227.32:4834 , piece_id=CBK2UQ252EPFQDTOYS2X5WAUTJN4L2DTHGIOJXFZXWBOLGPRETXQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.028430262 (173h38m51.579900723s) [noise=true , address=145.79.227.9:4366 , piece_id=YFWF2YL2YPYVMP5DHJKRUGVUUS2ALG4DRJ3HUZCDZM4N3N676FNQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.029687726 (173h38m51.581158177s) [noise=true , address=145.79.227.20:4438 , piece_id=6HY2HDNJAZ552ZKXFJ7VNTQNBAQU5ZQLNKAG2NZZWZQHN6EEDG6Q, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.031784128 (173h38m51.583254589s) [noise=true , address=45.91.116.5:4298 , piece_id=ARGPAZE6CTMM6UXSMPSVHDBEQWKVGKY7WXOVLPB7UH3JZYMDB7JQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.034100830 (173h38m51.585571291s) [noise=true , address=77.90.31.10:4518 , piece_id=MSKQ66KE4WANAE4AWOD2IKY3PQN2YGZ5N4W3NXGKXSOLE7B6H6WQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:54.850200282 (173h38m53.401670733s) [upload=152907]: Next returned nil segment Nov 16 13:42:54.850219118 (173h38m53.401689569s) [upload=152907]: Waiting for error group managing segments... Nov 16 13:42:55.154333032 (173h38m53.705803493s) [noise=true , address=45.91.116.7:4010 , piece_id=EO7IEOCWEJAMZZU4BLIB5ALFACWTWZE3QCXU3FUY4FKC4P444ORA, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:42:55.201689040 (173h38m53.753159501s) [noise=true , address=77.90.31.10:4518 , piece_id=MSKQ66KE4WANAE4AWOD2IKY3PQN2YGZ5N4W3NXGKXSOLE7B6H6WQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:42:55.232313345 (173h38m53.783783816s) [noise=true , address=45.91.116.5:4298 , piece_id=ARGPAZE6CTMM6UXSMPSVHDBEQWKVGKY7WXOVLPB7UH3JZYMDB7JQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:42:55.260281393 (173h38m53.811751854s) [noise=true , address=45.91.116.6:4974 , piece_id=ZTS3H7E6A3VJECLRRKRKSVX5PH3VSH7F2HAUPQSPH56NCI6RTFBQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:00.229127085 (173h38m58.780597546s) [noise=true , address=145.79.227.20:4438 , piece_id=6HY2HDNJAZ552ZKXFJ7VNTQNBAQU5ZQLNKAG2NZZWZQHN6EEDG6Q, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:00.358241119 (173h38m58.909711581s) [noise=true , address=145.79.227.32:4834 , piece_id=CBK2UQ252EPFQDTOYS2X5WAUTJN4L2DTHGIOJXFZXWBOLGPRETXQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:00.646958215 (173h38m59.198428676s) [noise=true , address=145.79.227.31:4798 , piece_id=DPLALMJC4JPC5D7BOK5SOSDFDJFDDPGSJWP4XDDQE22AMFNAKATQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:00.908824080 (173h38m59.460294541s) [noise=true , address=145.79.227.23:4547 , piece_id=P4WX2ZDHV2Z6BFMF6IACIYW4DXKYFVSULUTF7EK2EMRO5OA5X5RQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:01.155346527 (173h38m59.706816989s) [noise=true , address=45.91.116.14:4189 , piece_id=NUMS6Q3F2OUAQ2PSKE2CZK2WKRQLZUZPWVBA3UA6UNKDQACPYSJQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:01.839444794 (173h39m0.390915255s) [noise=true , address=145.79.227.11:4732 , piece_id=GR6SASHNUTHAEOUPK7DWKVPHRGMG72FQMF4RINJ5NTIQQGFCVZZQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:02.591249579 (173h39m1.14272004s) [noise=true , address=145.79.227.10:4402 , piece_id=DNT5QVAMRO5OMBNVTDRRUKACMM7M5472LD5E4ASKYIBKBNWYAVRA, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:02.616126740 (173h39m1.167597202s) [noise=true , address=145.79.227.31:4797 , piece_id=UKP3MHXBSALCWFIHCTSR5JFFXEWWAWGDS4JYUYS2JEREQQKTPJSA, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:02.734605482 (173h39m1.286075943s) [noise=true , address=145.79.227.27:4691 , piece_id=NSX5S2D7ZSA5I57B4EAQEQXEVUMPKNMRKRV3RHWGWJ2DN7SWUPGQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:02.734638004 (173h39m1.286108475s) [seg_pos={0 0}, upload=152907]: Segment reached optimal threshold of 13 pieces. Nov 16 13:43:02.735049748 (173h39m1.286520209s) [noise=true , address=145.79.227.9:4366 , piece_id=YFWF2YL2YPYVMP5DHJKRUGVUUS2ALG4DRJ3HUZCDZM4N3N676FNQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy): protocol: write tcp 188.166.225.170:60100->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:60100->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60100->145.79.227.9:4366: use of closed network connection Nov 16 13:43:02.735060949 (173h39m1.28653141s) [noise=true , address=145.79.227.9:4366 , piece_id=YFWF2YL2YPYVMP5DHJKRUGVUUS2ALG4DRJ3HUZCDZM4N3N676FNQ, seg_pos={0 0}, upload=152907]: Long tail cancellation due to optimal threshold Nov 16 13:43:02.735509193 (173h39m1.286979654s) [noise=true , address=77.90.31.14:4662 , piece_id=S5PPMCKIKSOCLUJ4CEH7ZDW3TVUZ7GTFFEIPCQK6FX25UIOUQHHQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): protocol: write tcp 188.166.225.170:50196->77.90.31.14:4662: use of closed network connection; write tcp 188.166.225.170:50196->77.90.31.14:4662: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50196->77.90.31.14:4662: use of closed network connection Nov 16 13:43:02.735518792 (173h39m1.286989253s) [noise=true , address=77.90.31.14:4662 , piece_id=S5PPMCKIKSOCLUJ4CEH7ZDW3TVUZ7GTFFEIPCQK6FX25UIOUQHHQ, seg_pos={0 0}, upload=152907]: Long tail cancellation due to optimal threshold Nov 16 13:43:02.736224335 (173h39m1.287694796s) [noise=true , address=145.79.227.20:4437 , piece_id=G5P6OYXVAAQDZKTPPHDNZLMYTT6L2TFWKH7N4F7L6742BWHOQGBQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV): protocol: read tcp 188.166.225.170:35300->145.79.227.20:4437: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 13:43:02.736239203 (173h39m1.287709654s) [noise=true , address=145.79.227.20:4437 , piece_id=G5P6OYXVAAQDZKTPPHDNZLMYTT6L2TFWKH7N4F7L6742BWHOQGBQ, seg_pos={0 0}, upload=152907]: Long tail cancellation due to optimal threshold Nov 16 13:43:02.736270833 (173h39m1.287741294s) [seg_pos={0 0}, upload=152907]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 16 13:43:02.736437060 (173h39m1.287907521s) [upload=152907]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 16 13:43:02.736444273 (173h39m1.287914735s) [upload=152907]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 16 13:43:05.031396865 (173h39m3.582867336s) [upload=152907]: Done uploading segments... Nov 16 13:43:05.031422193 (173h39m3.582892654s) [upload=152907]: Upload finished. err: <nil> Duration: 6.020916103s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 10 out of 60 is failed (threshold=6) [....X..XX....X......................X........X..X.....X..XX.] duration: 3.527585572s trace: https://jaeger.dev.storj.io/trace/16dd114e41a655be FAIL (2025-11-16 14:22:18 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.527614066s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 6.070711736s trace: https://jaeger.dev.storj.io/trace/3c77028f820d12f9 FAIL (2025-11-16 14:22:00 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, Piece ID: UKP3MHXBSALCWFIHCTSR5JFFXEWWAWGDS4JYUYS2JEREQQKTPJSA) context deadline exceeded; piecestore: (Node ID: 12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, Piece ID: DNT5QVAMRO5OMBNVTDRRUKACMM7M5472LD5E4ASKYIBKBNWYAVRA) context deadline exceeded; piecestore: (Node ID: 17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, Piece ID: DPLALMJC4JPC5D7BOK5SOSDFDJFDDPGSJWP4XDDQE22AMFNAKATQ) context deadline exceeded ======LOGS====== gment. Inline: false Nov 16 13:42:52.666975419 (173h38m51.21844588s) [upload=152907]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 16 13:42:52.666980539 (173h38m51.21845099s) [upload=152907]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 16 13:42:53.024947049 (173h38m51.5764175s) [seg_pos={0 0}, upload=152907]: Begin upload segment... Nov 16 13:42:53.025007074 (173h38m51.576477535s) [seg_pos={0 0}, upload=152907]: Done begin upload segment. Nov 16 13:42:53.025087707 (173h38m51.576558158s) [noise=true , address=77.90.31.14:4662 , piece_id=S5PPMCKIKSOCLUJ4CEH7ZDW3TVUZ7GTFFEIPCQK6FX25UIOUQHHQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.025098377 (173h38m51.576568828s) [noise=true , address=145.79.227.31:4797 , piece_id=UKP3MHXBSALCWFIHCTSR5JFFXEWWAWGDS4JYUYS2JEREQQKTPJSA, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.025961451 (173h38m51.577431912s) [noise=true , address=45.91.116.6:4974 , piece_id=ZTS3H7E6A3VJECLRRKRKSVX5PH3VSH7F2HAUPQSPH56NCI6RTFBQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.026101888 (173h38m51.577572339s) [noise=true , address=145.79.227.10:4402 , piece_id=DNT5QVAMRO5OMBNVTDRRUKACMM7M5472LD5E4ASKYIBKBNWYAVRA, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.026397721 (173h38m51.577868182s) [noise=true , address=145.79.227.11:4732 , piece_id=GR6SASHNUTHAEOUPK7DWKVPHRGMG72FQMF4RINJ5NTIQQGFCVZZQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.026543650 (173h38m51.578014111s) [noise=true , address=145.79.227.20:4437 , piece_id=G5P6OYXVAAQDZKTPPHDNZLMYTT6L2TFWKH7N4F7L6742BWHOQGBQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.026887915 (173h38m51.578358366s) [noise=true , address=45.91.116.14:4189 , piece_id=NUMS6Q3F2OUAQ2PSKE2CZK2WKRQLZUZPWVBA3UA6UNKDQACPYSJQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.027039624 (173h38m51.578510085s) [noise=true , address=145.79.227.27:4691 , piece_id=NSX5S2D7ZSA5I57B4EAQEQXEVUMPKNMRKRV3RHWGWJ2DN7SWUPGQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.027374482 (173h38m51.578844973s) [noise=true , address=145.79.227.31:4798 , piece_id=DPLALMJC4JPC5D7BOK5SOSDFDJFDDPGSJWP4XDDQE22AMFNAKATQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.027472337 (173h38m51.578942798s) [noise=true , address=45.91.116.7:4010 , piece_id=EO7IEOCWEJAMZZU4BLIB5ALFACWTWZE3QCXU3FUY4FKC4P444ORA, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.027920821 (173h38m51.579391283s) [noise=true , address=145.79.227.23:4547 , piece_id=P4WX2ZDHV2Z6BFMF6IACIYW4DXKYFVSULUTF7EK2EMRO5OA5X5RQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.028194272 (173h38m51.579664743s) [noise=true , address=145.79.227.32:4834 , piece_id=CBK2UQ252EPFQDTOYS2X5WAUTJN4L2DTHGIOJXFZXWBOLGPRETXQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.028430262 (173h38m51.579900723s) [noise=true , address=145.79.227.9:4366 , piece_id=YFWF2YL2YPYVMP5DHJKRUGVUUS2ALG4DRJ3HUZCDZM4N3N676FNQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.029687726 (173h38m51.581158177s) [noise=true , address=145.79.227.20:4438 , piece_id=6HY2HDNJAZ552ZKXFJ7VNTQNBAQU5ZQLNKAG2NZZWZQHN6EEDG6Q, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.031784128 (173h38m51.583254589s) [noise=true , address=45.91.116.5:4298 , piece_id=ARGPAZE6CTMM6UXSMPSVHDBEQWKVGKY7WXOVLPB7UH3JZYMDB7JQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.034100830 (173h38m51.585571291s) [noise=true , address=77.90.31.10:4518 , piece_id=MSKQ66KE4WANAE4AWOD2IKY3PQN2YGZ5N4W3NXGKXSOLE7B6H6WQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:54.850200282 (173h38m53.401670733s) [upload=152907]: Next returned nil segment Nov 16 13:42:54.850219118 (173h38m53.401689569s) [upload=152907]: Waiting for error group managing segments... Nov 16 13:42:55.154333032 (173h38m53.705803493s) [noise=true , address=45.91.116.7:4010 , piece_id=EO7IEOCWEJAMZZU4BLIB5ALFACWTWZE3QCXU3FUY4FKC4P444ORA, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:42:55.201689040 (173h38m53.753159501s) [noise=true , address=77.90.31.10:4518 , piece_id=MSKQ66KE4WANAE4AWOD2IKY3PQN2YGZ5N4W3NXGKXSOLE7B6H6WQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:42:55.232313345 (173h38m53.783783816s) [noise=true , address=45.91.116.5:4298 , piece_id=ARGPAZE6CTMM6UXSMPSVHDBEQWKVGKY7WXOVLPB7UH3JZYMDB7JQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:42:55.260281393 (173h38m53.811751854s) [noise=true , address=45.91.116.6:4974 , piece_id=ZTS3H7E6A3VJECLRRKRKSVX5PH3VSH7F2HAUPQSPH56NCI6RTFBQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:00.229127085 (173h38m58.780597546s) [noise=true , address=145.79.227.20:4438 , piece_id=6HY2HDNJAZ552ZKXFJ7VNTQNBAQU5ZQLNKAG2NZZWZQHN6EEDG6Q, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:00.358241119 (173h38m58.909711581s) [noise=true , address=145.79.227.32:4834 , piece_id=CBK2UQ252EPFQDTOYS2X5WAUTJN4L2DTHGIOJXFZXWBOLGPRETXQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:00.646958215 (173h38m59.198428676s) [noise=true , address=145.79.227.31:4798 , piece_id=DPLALMJC4JPC5D7BOK5SOSDFDJFDDPGSJWP4XDDQE22AMFNAKATQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:00.908824080 (173h38m59.460294541s) [noise=true , address=145.79.227.23:4547 , piece_id=P4WX2ZDHV2Z6BFMF6IACIYW4DXKYFVSULUTF7EK2EMRO5OA5X5RQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:01.155346527 (173h38m59.706816989s) [noise=true , address=45.91.116.14:4189 , piece_id=NUMS6Q3F2OUAQ2PSKE2CZK2WKRQLZUZPWVBA3UA6UNKDQACPYSJQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:01.839444794 (173h39m0.390915255s) [noise=true , address=145.79.227.11:4732 , piece_id=GR6SASHNUTHAEOUPK7DWKVPHRGMG72FQMF4RINJ5NTIQQGFCVZZQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:02.591249579 (173h39m1.14272004s) [noise=true , address=145.79.227.10:4402 , piece_id=DNT5QVAMRO5OMBNVTDRRUKACMM7M5472LD5E4ASKYIBKBNWYAVRA, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:02.616126740 (173h39m1.167597202s) [noise=true , address=145.79.227.31:4797 , piece_id=UKP3MHXBSALCWFIHCTSR5JFFXEWWAWGDS4JYUYS2JEREQQKTPJSA, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:02.734605482 (173h39m1.286075943s) [noise=true , address=145.79.227.27:4691 , piece_id=NSX5S2D7ZSA5I57B4EAQEQXEVUMPKNMRKRV3RHWGWJ2DN7SWUPGQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:02.734638004 (173h39m1.286108475s) [seg_pos={0 0}, upload=152907]: Segment reached optimal threshold of 13 pieces. Nov 16 13:43:02.735049748 (173h39m1.286520209s) [noise=true , address=145.79.227.9:4366 , piece_id=YFWF2YL2YPYVMP5DHJKRUGVUUS2ALG4DRJ3HUZCDZM4N3N676FNQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy): protocol: write tcp 188.166.225.170:60100->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:60100->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60100->145.79.227.9:4366: use of closed network connection Nov 16 13:43:02.735060949 (173h39m1.28653141s) [noise=true , address=145.79.227.9:4366 , piece_id=YFWF2YL2YPYVMP5DHJKRUGVUUS2ALG4DRJ3HUZCDZM4N3N676FNQ, seg_pos={0 0}, upload=152907]: Long tail cancellation due to optimal threshold Nov 16 13:43:02.735509193 (173h39m1.286979654s) [noise=true , address=77.90.31.14:4662 , piece_id=S5PPMCKIKSOCLUJ4CEH7ZDW3TVUZ7GTFFEIPCQK6FX25UIOUQHHQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): protocol: write tcp 188.166.225.170:50196->77.90.31.14:4662: use of closed network connection; write tcp 188.166.225.170:50196->77.90.31.14:4662: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50196->77.90.31.14:4662: use of closed network connection Nov 16 13:43:02.735518792 (173h39m1.286989253s) [noise=true , address=77.90.31.14:4662 , piece_id=S5PPMCKIKSOCLUJ4CEH7ZDW3TVUZ7GTFFEIPCQK6FX25UIOUQHHQ, seg_pos={0 0}, upload=152907]: Long tail cancellation due to optimal threshold Nov 16 13:43:02.736224335 (173h39m1.287694796s) [noise=true , address=145.79.227.20:4437 , piece_id=G5P6OYXVAAQDZKTPPHDNZLMYTT6L2TFWKH7N4F7L6742BWHOQGBQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV): protocol: read tcp 188.166.225.170:35300->145.79.227.20:4437: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 13:43:02.736239203 (173h39m1.287709654s) [noise=true , address=145.79.227.20:4437 , piece_id=G5P6OYXVAAQDZKTPPHDNZLMYTT6L2TFWKH7N4F7L6742BWHOQGBQ, seg_pos={0 0}, upload=152907]: Long tail cancellation due to optimal threshold Nov 16 13:43:02.736270833 (173h39m1.287741294s) [seg_pos={0 0}, upload=152907]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 16 13:43:02.736437060 (173h39m1.287907521s) [upload=152907]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 16 13:43:02.736444273 (173h39m1.287914735s) [upload=152907]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 16 13:43:05.031396865 (173h39m3.582867336s) [upload=152907]: Done uploading segments... Nov 16 13:43:05.031422193 (173h39m3.582892654s) [upload=152907]: Upload finished. err: <nil> Duration: 6.070738568s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 10 out of 60 is failed (threshold=6) [.....X..XX....X......................X........X..X.....X..XX] duration: 5.002615768s trace: https://jaeger.dev.storj.io/trace/16dd114e41a655be FAIL (2025-11-16 14:21:34 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.002641296s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 6.052560886s trace: https://jaeger.dev.storj.io/trace/3c77028f820d12f9 FAIL (2025-11-16 14:21:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, Piece ID: UKP3MHXBSALCWFIHCTSR5JFFXEWWAWGDS4JYUYS2JEREQQKTPJSA) context deadline exceeded; piecestore: (Node ID: 12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, Piece ID: DNT5QVAMRO5OMBNVTDRRUKACMM7M5472LD5E4ASKYIBKBNWYAVRA) context deadline exceeded; piecestore: (Node ID: 17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, Piece ID: DPLALMJC4JPC5D7BOK5SOSDFDJFDDPGSJWP4XDDQE22AMFNAKATQ) context deadline exceeded ======LOGS====== gment. Inline: false Nov 16 13:42:52.666975419 (173h38m51.21844588s) [upload=152907]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 16 13:42:52.666980539 (173h38m51.21845099s) [upload=152907]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 16 13:42:53.024947049 (173h38m51.5764175s) [seg_pos={0 0}, upload=152907]: Begin upload segment... Nov 16 13:42:53.025007074 (173h38m51.576477535s) [seg_pos={0 0}, upload=152907]: Done begin upload segment. Nov 16 13:42:53.025087707 (173h38m51.576558158s) [noise=true , address=77.90.31.14:4662 , piece_id=S5PPMCKIKSOCLUJ4CEH7ZDW3TVUZ7GTFFEIPCQK6FX25UIOUQHHQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.025098377 (173h38m51.576568828s) [noise=true , address=145.79.227.31:4797 , piece_id=UKP3MHXBSALCWFIHCTSR5JFFXEWWAWGDS4JYUYS2JEREQQKTPJSA, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.025961451 (173h38m51.577431912s) [noise=true , address=45.91.116.6:4974 , piece_id=ZTS3H7E6A3VJECLRRKRKSVX5PH3VSH7F2HAUPQSPH56NCI6RTFBQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.026101888 (173h38m51.577572339s) [noise=true , address=145.79.227.10:4402 , piece_id=DNT5QVAMRO5OMBNVTDRRUKACMM7M5472LD5E4ASKYIBKBNWYAVRA, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.026397721 (173h38m51.577868182s) [noise=true , address=145.79.227.11:4732 , piece_id=GR6SASHNUTHAEOUPK7DWKVPHRGMG72FQMF4RINJ5NTIQQGFCVZZQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.026543650 (173h38m51.578014111s) [noise=true , address=145.79.227.20:4437 , piece_id=G5P6OYXVAAQDZKTPPHDNZLMYTT6L2TFWKH7N4F7L6742BWHOQGBQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.026887915 (173h38m51.578358366s) [noise=true , address=45.91.116.14:4189 , piece_id=NUMS6Q3F2OUAQ2PSKE2CZK2WKRQLZUZPWVBA3UA6UNKDQACPYSJQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.027039624 (173h38m51.578510085s) [noise=true , address=145.79.227.27:4691 , piece_id=NSX5S2D7ZSA5I57B4EAQEQXEVUMPKNMRKRV3RHWGWJ2DN7SWUPGQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.027374482 (173h38m51.578844973s) [noise=true , address=145.79.227.31:4798 , piece_id=DPLALMJC4JPC5D7BOK5SOSDFDJFDDPGSJWP4XDDQE22AMFNAKATQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.027472337 (173h38m51.578942798s) [noise=true , address=45.91.116.7:4010 , piece_id=EO7IEOCWEJAMZZU4BLIB5ALFACWTWZE3QCXU3FUY4FKC4P444ORA, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.027920821 (173h38m51.579391283s) [noise=true , address=145.79.227.23:4547 , piece_id=P4WX2ZDHV2Z6BFMF6IACIYW4DXKYFVSULUTF7EK2EMRO5OA5X5RQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.028194272 (173h38m51.579664743s) [noise=true , address=145.79.227.32:4834 , piece_id=CBK2UQ252EPFQDTOYS2X5WAUTJN4L2DTHGIOJXFZXWBOLGPRETXQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.028430262 (173h38m51.579900723s) [noise=true , address=145.79.227.9:4366 , piece_id=YFWF2YL2YPYVMP5DHJKRUGVUUS2ALG4DRJ3HUZCDZM4N3N676FNQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.029687726 (173h38m51.581158177s) [noise=true , address=145.79.227.20:4438 , piece_id=6HY2HDNJAZ552ZKXFJ7VNTQNBAQU5ZQLNKAG2NZZWZQHN6EEDG6Q, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.031784128 (173h38m51.583254589s) [noise=true , address=45.91.116.5:4298 , piece_id=ARGPAZE6CTMM6UXSMPSVHDBEQWKVGKY7WXOVLPB7UH3JZYMDB7JQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.034100830 (173h38m51.585571291s) [noise=true , address=77.90.31.10:4518 , piece_id=MSKQ66KE4WANAE4AWOD2IKY3PQN2YGZ5N4W3NXGKXSOLE7B6H6WQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:54.850200282 (173h38m53.401670733s) [upload=152907]: Next returned nil segment Nov 16 13:42:54.850219118 (173h38m53.401689569s) [upload=152907]: Waiting for error group managing segments... Nov 16 13:42:55.154333032 (173h38m53.705803493s) [noise=true , address=45.91.116.7:4010 , piece_id=EO7IEOCWEJAMZZU4BLIB5ALFACWTWZE3QCXU3FUY4FKC4P444ORA, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:42:55.201689040 (173h38m53.753159501s) [noise=true , address=77.90.31.10:4518 , piece_id=MSKQ66KE4WANAE4AWOD2IKY3PQN2YGZ5N4W3NXGKXSOLE7B6H6WQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:42:55.232313345 (173h38m53.783783816s) [noise=true , address=45.91.116.5:4298 , piece_id=ARGPAZE6CTMM6UXSMPSVHDBEQWKVGKY7WXOVLPB7UH3JZYMDB7JQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:42:55.260281393 (173h38m53.811751854s) [noise=true , address=45.91.116.6:4974 , piece_id=ZTS3H7E6A3VJECLRRKRKSVX5PH3VSH7F2HAUPQSPH56NCI6RTFBQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:00.229127085 (173h38m58.780597546s) [noise=true , address=145.79.227.20:4438 , piece_id=6HY2HDNJAZ552ZKXFJ7VNTQNBAQU5ZQLNKAG2NZZWZQHN6EEDG6Q, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:00.358241119 (173h38m58.909711581s) [noise=true , address=145.79.227.32:4834 , piece_id=CBK2UQ252EPFQDTOYS2X5WAUTJN4L2DTHGIOJXFZXWBOLGPRETXQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:00.646958215 (173h38m59.198428676s) [noise=true , address=145.79.227.31:4798 , piece_id=DPLALMJC4JPC5D7BOK5SOSDFDJFDDPGSJWP4XDDQE22AMFNAKATQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:00.908824080 (173h38m59.460294541s) [noise=true , address=145.79.227.23:4547 , piece_id=P4WX2ZDHV2Z6BFMF6IACIYW4DXKYFVSULUTF7EK2EMRO5OA5X5RQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:01.155346527 (173h38m59.706816989s) [noise=true , address=45.91.116.14:4189 , piece_id=NUMS6Q3F2OUAQ2PSKE2CZK2WKRQLZUZPWVBA3UA6UNKDQACPYSJQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:01.839444794 (173h39m0.390915255s) [noise=true , address=145.79.227.11:4732 , piece_id=GR6SASHNUTHAEOUPK7DWKVPHRGMG72FQMF4RINJ5NTIQQGFCVZZQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:02.591249579 (173h39m1.14272004s) [noise=true , address=145.79.227.10:4402 , piece_id=DNT5QVAMRO5OMBNVTDRRUKACMM7M5472LD5E4ASKYIBKBNWYAVRA, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:02.616126740 (173h39m1.167597202s) [noise=true , address=145.79.227.31:4797 , piece_id=UKP3MHXBSALCWFIHCTSR5JFFXEWWAWGDS4JYUYS2JEREQQKTPJSA, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:02.734605482 (173h39m1.286075943s) [noise=true , address=145.79.227.27:4691 , piece_id=NSX5S2D7ZSA5I57B4EAQEQXEVUMPKNMRKRV3RHWGWJ2DN7SWUPGQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:02.734638004 (173h39m1.286108475s) [seg_pos={0 0}, upload=152907]: Segment reached optimal threshold of 13 pieces. Nov 16 13:43:02.735049748 (173h39m1.286520209s) [noise=true , address=145.79.227.9:4366 , piece_id=YFWF2YL2YPYVMP5DHJKRUGVUUS2ALG4DRJ3HUZCDZM4N3N676FNQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy): protocol: write tcp 188.166.225.170:60100->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:60100->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60100->145.79.227.9:4366: use of closed network connection Nov 16 13:43:02.735060949 (173h39m1.28653141s) [noise=true , address=145.79.227.9:4366 , piece_id=YFWF2YL2YPYVMP5DHJKRUGVUUS2ALG4DRJ3HUZCDZM4N3N676FNQ, seg_pos={0 0}, upload=152907]: Long tail cancellation due to optimal threshold Nov 16 13:43:02.735509193 (173h39m1.286979654s) [noise=true , address=77.90.31.14:4662 , piece_id=S5PPMCKIKSOCLUJ4CEH7ZDW3TVUZ7GTFFEIPCQK6FX25UIOUQHHQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): protocol: write tcp 188.166.225.170:50196->77.90.31.14:4662: use of closed network connection; write tcp 188.166.225.170:50196->77.90.31.14:4662: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50196->77.90.31.14:4662: use of closed network connection Nov 16 13:43:02.735518792 (173h39m1.286989253s) [noise=true , address=77.90.31.14:4662 , piece_id=S5PPMCKIKSOCLUJ4CEH7ZDW3TVUZ7GTFFEIPCQK6FX25UIOUQHHQ, seg_pos={0 0}, upload=152907]: Long tail cancellation due to optimal threshold Nov 16 13:43:02.736224335 (173h39m1.287694796s) [noise=true , address=145.79.227.20:4437 , piece_id=G5P6OYXVAAQDZKTPPHDNZLMYTT6L2TFWKH7N4F7L6742BWHOQGBQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV): protocol: read tcp 188.166.225.170:35300->145.79.227.20:4437: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 13:43:02.736239203 (173h39m1.287709654s) [noise=true , address=145.79.227.20:4437 , piece_id=G5P6OYXVAAQDZKTPPHDNZLMYTT6L2TFWKH7N4F7L6742BWHOQGBQ, seg_pos={0 0}, upload=152907]: Long tail cancellation due to optimal threshold Nov 16 13:43:02.736270833 (173h39m1.287741294s) [seg_pos={0 0}, upload=152907]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 16 13:43:02.736437060 (173h39m1.287907521s) [upload=152907]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 16 13:43:02.736444273 (173h39m1.287914735s) [upload=152907]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 16 13:43:05.031396865 (173h39m3.582867336s) [upload=152907]: Done uploading segments... Nov 16 13:43:05.031422193 (173h39m3.582892654s) [upload=152907]: Upload finished. err: <nil> Duration: 6.052609057s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 9 out of 60 is failed (threshold=6) [......X..XX....X......................X........X..X.....X..X] duration: 5.001477038s trace: https://jaeger.dev.storj.io/trace/5187960fc66f3d36 FAIL (2025-11-16 14:20:53 UTC) orbiter warning: context deadline exceeded Duration: 5.001503076s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 5.942602738s trace: https://jaeger.dev.storj.io/trace/3c77028f820d12f9 FAIL (2025-11-16 14:20:36 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, Piece ID: UKP3MHXBSALCWFIHCTSR5JFFXEWWAWGDS4JYUYS2JEREQQKTPJSA) context deadline exceeded; piecestore: (Node ID: 12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, Piece ID: DNT5QVAMRO5OMBNVTDRRUKACMM7M5472LD5E4ASKYIBKBNWYAVRA) context deadline exceeded; piecestore: (Node ID: 17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, Piece ID: DPLALMJC4JPC5D7BOK5SOSDFDJFDDPGSJWP4XDDQE22AMFNAKATQ) context deadline exceeded ======LOGS====== gment. Inline: false Nov 16 13:42:52.666975419 (173h38m51.21844588s) [upload=152907]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 16 13:42:52.666980539 (173h38m51.21845099s) [upload=152907]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 16 13:42:53.024947049 (173h38m51.5764175s) [seg_pos={0 0}, upload=152907]: Begin upload segment... Nov 16 13:42:53.025007074 (173h38m51.576477535s) [seg_pos={0 0}, upload=152907]: Done begin upload segment. Nov 16 13:42:53.025087707 (173h38m51.576558158s) [noise=true , address=77.90.31.14:4662 , piece_id=S5PPMCKIKSOCLUJ4CEH7ZDW3TVUZ7GTFFEIPCQK6FX25UIOUQHHQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.025098377 (173h38m51.576568828s) [noise=true , address=145.79.227.31:4797 , piece_id=UKP3MHXBSALCWFIHCTSR5JFFXEWWAWGDS4JYUYS2JEREQQKTPJSA, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.025961451 (173h38m51.577431912s) [noise=true , address=45.91.116.6:4974 , piece_id=ZTS3H7E6A3VJECLRRKRKSVX5PH3VSH7F2HAUPQSPH56NCI6RTFBQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.026101888 (173h38m51.577572339s) [noise=true , address=145.79.227.10:4402 , piece_id=DNT5QVAMRO5OMBNVTDRRUKACMM7M5472LD5E4ASKYIBKBNWYAVRA, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.026397721 (173h38m51.577868182s) [noise=true , address=145.79.227.11:4732 , piece_id=GR6SASHNUTHAEOUPK7DWKVPHRGMG72FQMF4RINJ5NTIQQGFCVZZQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.026543650 (173h38m51.578014111s) [noise=true , address=145.79.227.20:4437 , piece_id=G5P6OYXVAAQDZKTPPHDNZLMYTT6L2TFWKH7N4F7L6742BWHOQGBQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.026887915 (173h38m51.578358366s) [noise=true , address=45.91.116.14:4189 , piece_id=NUMS6Q3F2OUAQ2PSKE2CZK2WKRQLZUZPWVBA3UA6UNKDQACPYSJQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.027039624 (173h38m51.578510085s) [noise=true , address=145.79.227.27:4691 , piece_id=NSX5S2D7ZSA5I57B4EAQEQXEVUMPKNMRKRV3RHWGWJ2DN7SWUPGQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.027374482 (173h38m51.578844973s) [noise=true , address=145.79.227.31:4798 , piece_id=DPLALMJC4JPC5D7BOK5SOSDFDJFDDPGSJWP4XDDQE22AMFNAKATQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.027472337 (173h38m51.578942798s) [noise=true , address=45.91.116.7:4010 , piece_id=EO7IEOCWEJAMZZU4BLIB5ALFACWTWZE3QCXU3FUY4FKC4P444ORA, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.027920821 (173h38m51.579391283s) [noise=true , address=145.79.227.23:4547 , piece_id=P4WX2ZDHV2Z6BFMF6IACIYW4DXKYFVSULUTF7EK2EMRO5OA5X5RQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.028194272 (173h38m51.579664743s) [noise=true , address=145.79.227.32:4834 , piece_id=CBK2UQ252EPFQDTOYS2X5WAUTJN4L2DTHGIOJXFZXWBOLGPRETXQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.028430262 (173h38m51.579900723s) [noise=true , address=145.79.227.9:4366 , piece_id=YFWF2YL2YPYVMP5DHJKRUGVUUS2ALG4DRJ3HUZCDZM4N3N676FNQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.029687726 (173h38m51.581158177s) [noise=true , address=145.79.227.20:4438 , piece_id=6HY2HDNJAZ552ZKXFJ7VNTQNBAQU5ZQLNKAG2NZZWZQHN6EEDG6Q, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.031784128 (173h38m51.583254589s) [noise=true , address=45.91.116.5:4298 , piece_id=ARGPAZE6CTMM6UXSMPSVHDBEQWKVGKY7WXOVLPB7UH3JZYMDB7JQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.034100830 (173h38m51.585571291s) [noise=true , address=77.90.31.10:4518 , piece_id=MSKQ66KE4WANAE4AWOD2IKY3PQN2YGZ5N4W3NXGKXSOLE7B6H6WQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:54.850200282 (173h38m53.401670733s) [upload=152907]: Next returned nil segment Nov 16 13:42:54.850219118 (173h38m53.401689569s) [upload=152907]: Waiting for error group managing segments... Nov 16 13:42:55.154333032 (173h38m53.705803493s) [noise=true , address=45.91.116.7:4010 , piece_id=EO7IEOCWEJAMZZU4BLIB5ALFACWTWZE3QCXU3FUY4FKC4P444ORA, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:42:55.201689040 (173h38m53.753159501s) [noise=true , address=77.90.31.10:4518 , piece_id=MSKQ66KE4WANAE4AWOD2IKY3PQN2YGZ5N4W3NXGKXSOLE7B6H6WQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:42:55.232313345 (173h38m53.783783816s) [noise=true , address=45.91.116.5:4298 , piece_id=ARGPAZE6CTMM6UXSMPSVHDBEQWKVGKY7WXOVLPB7UH3JZYMDB7JQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:42:55.260281393 (173h38m53.811751854s) [noise=true , address=45.91.116.6:4974 , piece_id=ZTS3H7E6A3VJECLRRKRKSVX5PH3VSH7F2HAUPQSPH56NCI6RTFBQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:00.229127085 (173h38m58.780597546s) [noise=true , address=145.79.227.20:4438 , piece_id=6HY2HDNJAZ552ZKXFJ7VNTQNBAQU5ZQLNKAG2NZZWZQHN6EEDG6Q, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:00.358241119 (173h38m58.909711581s) [noise=true , address=145.79.227.32:4834 , piece_id=CBK2UQ252EPFQDTOYS2X5WAUTJN4L2DTHGIOJXFZXWBOLGPRETXQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:00.646958215 (173h38m59.198428676s) [noise=true , address=145.79.227.31:4798 , piece_id=DPLALMJC4JPC5D7BOK5SOSDFDJFDDPGSJWP4XDDQE22AMFNAKATQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:00.908824080 (173h38m59.460294541s) [noise=true , address=145.79.227.23:4547 , piece_id=P4WX2ZDHV2Z6BFMF6IACIYW4DXKYFVSULUTF7EK2EMRO5OA5X5RQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:01.155346527 (173h38m59.706816989s) [noise=true , address=45.91.116.14:4189 , piece_id=NUMS6Q3F2OUAQ2PSKE2CZK2WKRQLZUZPWVBA3UA6UNKDQACPYSJQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:01.839444794 (173h39m0.390915255s) [noise=true , address=145.79.227.11:4732 , piece_id=GR6SASHNUTHAEOUPK7DWKVPHRGMG72FQMF4RINJ5NTIQQGFCVZZQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:02.591249579 (173h39m1.14272004s) [noise=true , address=145.79.227.10:4402 , piece_id=DNT5QVAMRO5OMBNVTDRRUKACMM7M5472LD5E4ASKYIBKBNWYAVRA, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:02.616126740 (173h39m1.167597202s) [noise=true , address=145.79.227.31:4797 , piece_id=UKP3MHXBSALCWFIHCTSR5JFFXEWWAWGDS4JYUYS2JEREQQKTPJSA, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:02.734605482 (173h39m1.286075943s) [noise=true , address=145.79.227.27:4691 , piece_id=NSX5S2D7ZSA5I57B4EAQEQXEVUMPKNMRKRV3RHWGWJ2DN7SWUPGQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:02.734638004 (173h39m1.286108475s) [seg_pos={0 0}, upload=152907]: Segment reached optimal threshold of 13 pieces. Nov 16 13:43:02.735049748 (173h39m1.286520209s) [noise=true , address=145.79.227.9:4366 , piece_id=YFWF2YL2YPYVMP5DHJKRUGVUUS2ALG4DRJ3HUZCDZM4N3N676FNQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy): protocol: write tcp 188.166.225.170:60100->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:60100->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60100->145.79.227.9:4366: use of closed network connection Nov 16 13:43:02.735060949 (173h39m1.28653141s) [noise=true , address=145.79.227.9:4366 , piece_id=YFWF2YL2YPYVMP5DHJKRUGVUUS2ALG4DRJ3HUZCDZM4N3N676FNQ, seg_pos={0 0}, upload=152907]: Long tail cancellation due to optimal threshold Nov 16 13:43:02.735509193 (173h39m1.286979654s) [noise=true , address=77.90.31.14:4662 , piece_id=S5PPMCKIKSOCLUJ4CEH7ZDW3TVUZ7GTFFEIPCQK6FX25UIOUQHHQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): protocol: write tcp 188.166.225.170:50196->77.90.31.14:4662: use of closed network connection; write tcp 188.166.225.170:50196->77.90.31.14:4662: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50196->77.90.31.14:4662: use of closed network connection Nov 16 13:43:02.735518792 (173h39m1.286989253s) [noise=true , address=77.90.31.14:4662 , piece_id=S5PPMCKIKSOCLUJ4CEH7ZDW3TVUZ7GTFFEIPCQK6FX25UIOUQHHQ, seg_pos={0 0}, upload=152907]: Long tail cancellation due to optimal threshold Nov 16 13:43:02.736224335 (173h39m1.287694796s) [noise=true , address=145.79.227.20:4437 , piece_id=G5P6OYXVAAQDZKTPPHDNZLMYTT6L2TFWKH7N4F7L6742BWHOQGBQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV): protocol: read tcp 188.166.225.170:35300->145.79.227.20:4437: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 13:43:02.736239203 (173h39m1.287709654s) [noise=true , address=145.79.227.20:4437 , piece_id=G5P6OYXVAAQDZKTPPHDNZLMYTT6L2TFWKH7N4F7L6742BWHOQGBQ, seg_pos={0 0}, upload=152907]: Long tail cancellation due to optimal threshold Nov 16 13:43:02.736270833 (173h39m1.287741294s) [seg_pos={0 0}, upload=152907]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 16 13:43:02.736437060 (173h39m1.287907521s) [upload=152907]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 16 13:43:02.736444273 (173h39m1.287914735s) [upload=152907]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 16 13:43:05.031396865 (173h39m3.582867336s) [upload=152907]: Done uploading segments... Nov 16 13:43:05.031422193 (173h39m3.582892654s) [upload=152907]: Upload finished. err: <nil> Duration: 5.942654747s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 6.198838495s trace: https://jaeger.dev.storj.io/trace/3c77028f820d12f9 FAIL (2025-11-16 14:20:14 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, Piece ID: UKP3MHXBSALCWFIHCTSR5JFFXEWWAWGDS4JYUYS2JEREQQKTPJSA) context deadline exceeded; piecestore: (Node ID: 12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, Piece ID: DNT5QVAMRO5OMBNVTDRRUKACMM7M5472LD5E4ASKYIBKBNWYAVRA) context deadline exceeded; piecestore: (Node ID: 17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, Piece ID: DPLALMJC4JPC5D7BOK5SOSDFDJFDDPGSJWP4XDDQE22AMFNAKATQ) context deadline exceeded ======LOGS====== gment. Inline: false Nov 16 13:42:52.666975419 (173h38m51.21844588s) [upload=152907]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 16 13:42:52.666980539 (173h38m51.21845099s) [upload=152907]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 16 13:42:53.024947049 (173h38m51.5764175s) [seg_pos={0 0}, upload=152907]: Begin upload segment... Nov 16 13:42:53.025007074 (173h38m51.576477535s) [seg_pos={0 0}, upload=152907]: Done begin upload segment. Nov 16 13:42:53.025087707 (173h38m51.576558158s) [noise=true , address=77.90.31.14:4662 , piece_id=S5PPMCKIKSOCLUJ4CEH7ZDW3TVUZ7GTFFEIPCQK6FX25UIOUQHHQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.025098377 (173h38m51.576568828s) [noise=true , address=145.79.227.31:4797 , piece_id=UKP3MHXBSALCWFIHCTSR5JFFXEWWAWGDS4JYUYS2JEREQQKTPJSA, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.025961451 (173h38m51.577431912s) [noise=true , address=45.91.116.6:4974 , piece_id=ZTS3H7E6A3VJECLRRKRKSVX5PH3VSH7F2HAUPQSPH56NCI6RTFBQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.026101888 (173h38m51.577572339s) [noise=true , address=145.79.227.10:4402 , piece_id=DNT5QVAMRO5OMBNVTDRRUKACMM7M5472LD5E4ASKYIBKBNWYAVRA, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.026397721 (173h38m51.577868182s) [noise=true , address=145.79.227.11:4732 , piece_id=GR6SASHNUTHAEOUPK7DWKVPHRGMG72FQMF4RINJ5NTIQQGFCVZZQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.026543650 (173h38m51.578014111s) [noise=true , address=145.79.227.20:4437 , piece_id=G5P6OYXVAAQDZKTPPHDNZLMYTT6L2TFWKH7N4F7L6742BWHOQGBQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.026887915 (173h38m51.578358366s) [noise=true , address=45.91.116.14:4189 , piece_id=NUMS6Q3F2OUAQ2PSKE2CZK2WKRQLZUZPWVBA3UA6UNKDQACPYSJQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.027039624 (173h38m51.578510085s) [noise=true , address=145.79.227.27:4691 , piece_id=NSX5S2D7ZSA5I57B4EAQEQXEVUMPKNMRKRV3RHWGWJ2DN7SWUPGQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.027374482 (173h38m51.578844973s) [noise=true , address=145.79.227.31:4798 , piece_id=DPLALMJC4JPC5D7BOK5SOSDFDJFDDPGSJWP4XDDQE22AMFNAKATQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.027472337 (173h38m51.578942798s) [noise=true , address=45.91.116.7:4010 , piece_id=EO7IEOCWEJAMZZU4BLIB5ALFACWTWZE3QCXU3FUY4FKC4P444ORA, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.027920821 (173h38m51.579391283s) [noise=true , address=145.79.227.23:4547 , piece_id=P4WX2ZDHV2Z6BFMF6IACIYW4DXKYFVSULUTF7EK2EMRO5OA5X5RQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.028194272 (173h38m51.579664743s) [noise=true , address=145.79.227.32:4834 , piece_id=CBK2UQ252EPFQDTOYS2X5WAUTJN4L2DTHGIOJXFZXWBOLGPRETXQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.028430262 (173h38m51.579900723s) [noise=true , address=145.79.227.9:4366 , piece_id=YFWF2YL2YPYVMP5DHJKRUGVUUS2ALG4DRJ3HUZCDZM4N3N676FNQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.029687726 (173h38m51.581158177s) [noise=true , address=145.79.227.20:4438 , piece_id=6HY2HDNJAZ552ZKXFJ7VNTQNBAQU5ZQLNKAG2NZZWZQHN6EEDG6Q, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.031784128 (173h38m51.583254589s) [noise=true , address=45.91.116.5:4298 , piece_id=ARGPAZE6CTMM6UXSMPSVHDBEQWKVGKY7WXOVLPB7UH3JZYMDB7JQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.034100830 (173h38m51.585571291s) [noise=true , address=77.90.31.10:4518 , piece_id=MSKQ66KE4WANAE4AWOD2IKY3PQN2YGZ5N4W3NXGKXSOLE7B6H6WQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:54.850200282 (173h38m53.401670733s) [upload=152907]: Next returned nil segment Nov 16 13:42:54.850219118 (173h38m53.401689569s) [upload=152907]: Waiting for error group managing segments... Nov 16 13:42:55.154333032 (173h38m53.705803493s) [noise=true , address=45.91.116.7:4010 , piece_id=EO7IEOCWEJAMZZU4BLIB5ALFACWTWZE3QCXU3FUY4FKC4P444ORA, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:42:55.201689040 (173h38m53.753159501s) [noise=true , address=77.90.31.10:4518 , piece_id=MSKQ66KE4WANAE4AWOD2IKY3PQN2YGZ5N4W3NXGKXSOLE7B6H6WQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:42:55.232313345 (173h38m53.783783816s) [noise=true , address=45.91.116.5:4298 , piece_id=ARGPAZE6CTMM6UXSMPSVHDBEQWKVGKY7WXOVLPB7UH3JZYMDB7JQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:42:55.260281393 (173h38m53.811751854s) [noise=true , address=45.91.116.6:4974 , piece_id=ZTS3H7E6A3VJECLRRKRKSVX5PH3VSH7F2HAUPQSPH56NCI6RTFBQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:00.229127085 (173h38m58.780597546s) [noise=true , address=145.79.227.20:4438 , piece_id=6HY2HDNJAZ552ZKXFJ7VNTQNBAQU5ZQLNKAG2NZZWZQHN6EEDG6Q, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:00.358241119 (173h38m58.909711581s) [noise=true , address=145.79.227.32:4834 , piece_id=CBK2UQ252EPFQDTOYS2X5WAUTJN4L2DTHGIOJXFZXWBOLGPRETXQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:00.646958215 (173h38m59.198428676s) [noise=true , address=145.79.227.31:4798 , piece_id=DPLALMJC4JPC5D7BOK5SOSDFDJFDDPGSJWP4XDDQE22AMFNAKATQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:00.908824080 (173h38m59.460294541s) [noise=true , address=145.79.227.23:4547 , piece_id=P4WX2ZDHV2Z6BFMF6IACIYW4DXKYFVSULUTF7EK2EMRO5OA5X5RQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:01.155346527 (173h38m59.706816989s) [noise=true , address=45.91.116.14:4189 , piece_id=NUMS6Q3F2OUAQ2PSKE2CZK2WKRQLZUZPWVBA3UA6UNKDQACPYSJQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:01.839444794 (173h39m0.390915255s) [noise=true , address=145.79.227.11:4732 , piece_id=GR6SASHNUTHAEOUPK7DWKVPHRGMG72FQMF4RINJ5NTIQQGFCVZZQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:02.591249579 (173h39m1.14272004s) [noise=true , address=145.79.227.10:4402 , piece_id=DNT5QVAMRO5OMBNVTDRRUKACMM7M5472LD5E4ASKYIBKBNWYAVRA, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:02.616126740 (173h39m1.167597202s) [noise=true , address=145.79.227.31:4797 , piece_id=UKP3MHXBSALCWFIHCTSR5JFFXEWWAWGDS4JYUYS2JEREQQKTPJSA, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:02.734605482 (173h39m1.286075943s) [noise=true , address=145.79.227.27:4691 , piece_id=NSX5S2D7ZSA5I57B4EAQEQXEVUMPKNMRKRV3RHWGWJ2DN7SWUPGQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:02.734638004 (173h39m1.286108475s) [seg_pos={0 0}, upload=152907]: Segment reached optimal threshold of 13 pieces. Nov 16 13:43:02.735049748 (173h39m1.286520209s) [noise=true , address=145.79.227.9:4366 , piece_id=YFWF2YL2YPYVMP5DHJKRUGVUUS2ALG4DRJ3HUZCDZM4N3N676FNQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy): protocol: write tcp 188.166.225.170:60100->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:60100->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60100->145.79.227.9:4366: use of closed network connection Nov 16 13:43:02.735060949 (173h39m1.28653141s) [noise=true , address=145.79.227.9:4366 , piece_id=YFWF2YL2YPYVMP5DHJKRUGVUUS2ALG4DRJ3HUZCDZM4N3N676FNQ, seg_pos={0 0}, upload=152907]: Long tail cancellation due to optimal threshold Nov 16 13:43:02.735509193 (173h39m1.286979654s) [noise=true , address=77.90.31.14:4662 , piece_id=S5PPMCKIKSOCLUJ4CEH7ZDW3TVUZ7GTFFEIPCQK6FX25UIOUQHHQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): protocol: write tcp 188.166.225.170:50196->77.90.31.14:4662: use of closed network connection; write tcp 188.166.225.170:50196->77.90.31.14:4662: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50196->77.90.31.14:4662: use of closed network connection Nov 16 13:43:02.735518792 (173h39m1.286989253s) [noise=true , address=77.90.31.14:4662 , piece_id=S5PPMCKIKSOCLUJ4CEH7ZDW3TVUZ7GTFFEIPCQK6FX25UIOUQHHQ, seg_pos={0 0}, upload=152907]: Long tail cancellation due to optimal threshold Nov 16 13:43:02.736224335 (173h39m1.287694796s) [noise=true , address=145.79.227.20:4437 , piece_id=G5P6OYXVAAQDZKTPPHDNZLMYTT6L2TFWKH7N4F7L6742BWHOQGBQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV): protocol: read tcp 188.166.225.170:35300->145.79.227.20:4437: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 13:43:02.736239203 (173h39m1.287709654s) [noise=true , address=145.79.227.20:4437 , piece_id=G5P6OYXVAAQDZKTPPHDNZLMYTT6L2TFWKH7N4F7L6742BWHOQGBQ, seg_pos={0 0}, upload=152907]: Long tail cancellation due to optimal threshold Nov 16 13:43:02.736270833 (173h39m1.287741294s) [seg_pos={0 0}, upload=152907]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 16 13:43:02.736437060 (173h39m1.287907521s) [upload=152907]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 16 13:43:02.736444273 (173h39m1.287914735s) [upload=152907]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 16 13:43:05.031396865 (173h39m3.582867336s) [upload=152907]: Done uploading segments... Nov 16 13:43:05.031422193 (173h39m3.582892654s) [upload=152907]: Upload finished. err: <nil> Duration: 6.198861207s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 9 out of 60 is failed (threshold=6) [X......X..XX....X......................X........X..X.....X..] duration: 4.239437172s trace: https://jaeger.dev.storj.io/trace/6d62f0753f4f7865 FAIL (2025-11-16 14:20:11 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.23946703s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 6.085479303s trace: https://jaeger.dev.storj.io/trace/3c77028f820d12f9 FAIL (2025-11-16 14:19:42 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, Piece ID: UKP3MHXBSALCWFIHCTSR5JFFXEWWAWGDS4JYUYS2JEREQQKTPJSA) context deadline exceeded; piecestore: (Node ID: 12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, Piece ID: DNT5QVAMRO5OMBNVTDRRUKACMM7M5472LD5E4ASKYIBKBNWYAVRA) context deadline exceeded; piecestore: (Node ID: 17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, Piece ID: DPLALMJC4JPC5D7BOK5SOSDFDJFDDPGSJWP4XDDQE22AMFNAKATQ) context deadline exceeded ======LOGS====== gment. Inline: false Nov 16 13:42:52.666975419 (173h38m51.21844588s) [upload=152907]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 16 13:42:52.666980539 (173h38m51.21845099s) [upload=152907]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 16 13:42:53.024947049 (173h38m51.5764175s) [seg_pos={0 0}, upload=152907]: Begin upload segment... Nov 16 13:42:53.025007074 (173h38m51.576477535s) [seg_pos={0 0}, upload=152907]: Done begin upload segment. Nov 16 13:42:53.025087707 (173h38m51.576558158s) [noise=true , address=77.90.31.14:4662 , piece_id=S5PPMCKIKSOCLUJ4CEH7ZDW3TVUZ7GTFFEIPCQK6FX25UIOUQHHQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.025098377 (173h38m51.576568828s) [noise=true , address=145.79.227.31:4797 , piece_id=UKP3MHXBSALCWFIHCTSR5JFFXEWWAWGDS4JYUYS2JEREQQKTPJSA, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.025961451 (173h38m51.577431912s) [noise=true , address=45.91.116.6:4974 , piece_id=ZTS3H7E6A3VJECLRRKRKSVX5PH3VSH7F2HAUPQSPH56NCI6RTFBQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.026101888 (173h38m51.577572339s) [noise=true , address=145.79.227.10:4402 , piece_id=DNT5QVAMRO5OMBNVTDRRUKACMM7M5472LD5E4ASKYIBKBNWYAVRA, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.026397721 (173h38m51.577868182s) [noise=true , address=145.79.227.11:4732 , piece_id=GR6SASHNUTHAEOUPK7DWKVPHRGMG72FQMF4RINJ5NTIQQGFCVZZQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.026543650 (173h38m51.578014111s) [noise=true , address=145.79.227.20:4437 , piece_id=G5P6OYXVAAQDZKTPPHDNZLMYTT6L2TFWKH7N4F7L6742BWHOQGBQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.026887915 (173h38m51.578358366s) [noise=true , address=45.91.116.14:4189 , piece_id=NUMS6Q3F2OUAQ2PSKE2CZK2WKRQLZUZPWVBA3UA6UNKDQACPYSJQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.027039624 (173h38m51.578510085s) [noise=true , address=145.79.227.27:4691 , piece_id=NSX5S2D7ZSA5I57B4EAQEQXEVUMPKNMRKRV3RHWGWJ2DN7SWUPGQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.027374482 (173h38m51.578844973s) [noise=true , address=145.79.227.31:4798 , piece_id=DPLALMJC4JPC5D7BOK5SOSDFDJFDDPGSJWP4XDDQE22AMFNAKATQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.027472337 (173h38m51.578942798s) [noise=true , address=45.91.116.7:4010 , piece_id=EO7IEOCWEJAMZZU4BLIB5ALFACWTWZE3QCXU3FUY4FKC4P444ORA, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.027920821 (173h38m51.579391283s) [noise=true , address=145.79.227.23:4547 , piece_id=P4WX2ZDHV2Z6BFMF6IACIYW4DXKYFVSULUTF7EK2EMRO5OA5X5RQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.028194272 (173h38m51.579664743s) [noise=true , address=145.79.227.32:4834 , piece_id=CBK2UQ252EPFQDTOYS2X5WAUTJN4L2DTHGIOJXFZXWBOLGPRETXQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.028430262 (173h38m51.579900723s) [noise=true , address=145.79.227.9:4366 , piece_id=YFWF2YL2YPYVMP5DHJKRUGVUUS2ALG4DRJ3HUZCDZM4N3N676FNQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.029687726 (173h38m51.581158177s) [noise=true , address=145.79.227.20:4438 , piece_id=6HY2HDNJAZ552ZKXFJ7VNTQNBAQU5ZQLNKAG2NZZWZQHN6EEDG6Q, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.031784128 (173h38m51.583254589s) [noise=true , address=45.91.116.5:4298 , piece_id=ARGPAZE6CTMM6UXSMPSVHDBEQWKVGKY7WXOVLPB7UH3JZYMDB7JQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.034100830 (173h38m51.585571291s) [noise=true , address=77.90.31.10:4518 , piece_id=MSKQ66KE4WANAE4AWOD2IKY3PQN2YGZ5N4W3NXGKXSOLE7B6H6WQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:54.850200282 (173h38m53.401670733s) [upload=152907]: Next returned nil segment Nov 16 13:42:54.850219118 (173h38m53.401689569s) [upload=152907]: Waiting for error group managing segments... Nov 16 13:42:55.154333032 (173h38m53.705803493s) [noise=true , address=45.91.116.7:4010 , piece_id=EO7IEOCWEJAMZZU4BLIB5ALFACWTWZE3QCXU3FUY4FKC4P444ORA, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:42:55.201689040 (173h38m53.753159501s) [noise=true , address=77.90.31.10:4518 , piece_id=MSKQ66KE4WANAE4AWOD2IKY3PQN2YGZ5N4W3NXGKXSOLE7B6H6WQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:42:55.232313345 (173h38m53.783783816s) [noise=true , address=45.91.116.5:4298 , piece_id=ARGPAZE6CTMM6UXSMPSVHDBEQWKVGKY7WXOVLPB7UH3JZYMDB7JQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:42:55.260281393 (173h38m53.811751854s) [noise=true , address=45.91.116.6:4974 , piece_id=ZTS3H7E6A3VJECLRRKRKSVX5PH3VSH7F2HAUPQSPH56NCI6RTFBQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:00.229127085 (173h38m58.780597546s) [noise=true , address=145.79.227.20:4438 , piece_id=6HY2HDNJAZ552ZKXFJ7VNTQNBAQU5ZQLNKAG2NZZWZQHN6EEDG6Q, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:00.358241119 (173h38m58.909711581s) [noise=true , address=145.79.227.32:4834 , piece_id=CBK2UQ252EPFQDTOYS2X5WAUTJN4L2DTHGIOJXFZXWBOLGPRETXQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:00.646958215 (173h38m59.198428676s) [noise=true , address=145.79.227.31:4798 , piece_id=DPLALMJC4JPC5D7BOK5SOSDFDJFDDPGSJWP4XDDQE22AMFNAKATQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:00.908824080 (173h38m59.460294541s) [noise=true , address=145.79.227.23:4547 , piece_id=P4WX2ZDHV2Z6BFMF6IACIYW4DXKYFVSULUTF7EK2EMRO5OA5X5RQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:01.155346527 (173h38m59.706816989s) [noise=true , address=45.91.116.14:4189 , piece_id=NUMS6Q3F2OUAQ2PSKE2CZK2WKRQLZUZPWVBA3UA6UNKDQACPYSJQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:01.839444794 (173h39m0.390915255s) [noise=true , address=145.79.227.11:4732 , piece_id=GR6SASHNUTHAEOUPK7DWKVPHRGMG72FQMF4RINJ5NTIQQGFCVZZQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:02.591249579 (173h39m1.14272004s) [noise=true , address=145.79.227.10:4402 , piece_id=DNT5QVAMRO5OMBNVTDRRUKACMM7M5472LD5E4ASKYIBKBNWYAVRA, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:02.616126740 (173h39m1.167597202s) [noise=true , address=145.79.227.31:4797 , piece_id=UKP3MHXBSALCWFIHCTSR5JFFXEWWAWGDS4JYUYS2JEREQQKTPJSA, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:02.734605482 (173h39m1.286075943s) [noise=true , address=145.79.227.27:4691 , piece_id=NSX5S2D7ZSA5I57B4EAQEQXEVUMPKNMRKRV3RHWGWJ2DN7SWUPGQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:02.734638004 (173h39m1.286108475s) [seg_pos={0 0}, upload=152907]: Segment reached optimal threshold of 13 pieces. Nov 16 13:43:02.735049748 (173h39m1.286520209s) [noise=true , address=145.79.227.9:4366 , piece_id=YFWF2YL2YPYVMP5DHJKRUGVUUS2ALG4DRJ3HUZCDZM4N3N676FNQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy): protocol: write tcp 188.166.225.170:60100->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:60100->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60100->145.79.227.9:4366: use of closed network connection Nov 16 13:43:02.735060949 (173h39m1.28653141s) [noise=true , address=145.79.227.9:4366 , piece_id=YFWF2YL2YPYVMP5DHJKRUGVUUS2ALG4DRJ3HUZCDZM4N3N676FNQ, seg_pos={0 0}, upload=152907]: Long tail cancellation due to optimal threshold Nov 16 13:43:02.735509193 (173h39m1.286979654s) [noise=true , address=77.90.31.14:4662 , piece_id=S5PPMCKIKSOCLUJ4CEH7ZDW3TVUZ7GTFFEIPCQK6FX25UIOUQHHQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): protocol: write tcp 188.166.225.170:50196->77.90.31.14:4662: use of closed network connection; write tcp 188.166.225.170:50196->77.90.31.14:4662: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50196->77.90.31.14:4662: use of closed network connection Nov 16 13:43:02.735518792 (173h39m1.286989253s) [noise=true , address=77.90.31.14:4662 , piece_id=S5PPMCKIKSOCLUJ4CEH7ZDW3TVUZ7GTFFEIPCQK6FX25UIOUQHHQ, seg_pos={0 0}, upload=152907]: Long tail cancellation due to optimal threshold Nov 16 13:43:02.736224335 (173h39m1.287694796s) [noise=true , address=145.79.227.20:4437 , piece_id=G5P6OYXVAAQDZKTPPHDNZLMYTT6L2TFWKH7N4F7L6742BWHOQGBQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV): protocol: read tcp 188.166.225.170:35300->145.79.227.20:4437: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 13:43:02.736239203 (173h39m1.287709654s) [noise=true , address=145.79.227.20:4437 , piece_id=G5P6OYXVAAQDZKTPPHDNZLMYTT6L2TFWKH7N4F7L6742BWHOQGBQ, seg_pos={0 0}, upload=152907]: Long tail cancellation due to optimal threshold Nov 16 13:43:02.736270833 (173h39m1.287741294s) [seg_pos={0 0}, upload=152907]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 16 13:43:02.736437060 (173h39m1.287907521s) [upload=152907]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 16 13:43:02.736444273 (173h39m1.287914735s) [upload=152907]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 16 13:43:05.031396865 (173h39m3.582867336s) [upload=152907]: Done uploading segments... Nov 16 13:43:05.031422193 (173h39m3.582892654s) [upload=152907]: Upload finished. err: <nil> Duration: 6.085524149s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 9 out of 60 is failed (threshold=6) [.X......X..XX....X......................X........X..X.....X.] duration: 3.542651911s trace: https://jaeger.dev.storj.io/trace/6d62f0753f4f7865 FAIL (2025-11-16 14:19:05 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.542688411s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 7.944293109s trace: https://jaeger.dev.storj.io/trace/3c77028f820d12f9 FAIL (2025-11-16 14:18:51 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, Piece ID: UKP3MHXBSALCWFIHCTSR5JFFXEWWAWGDS4JYUYS2JEREQQKTPJSA) context deadline exceeded; piecestore: (Node ID: 12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, Piece ID: DNT5QVAMRO5OMBNVTDRRUKACMM7M5472LD5E4ASKYIBKBNWYAVRA) context deadline exceeded; piecestore: (Node ID: 17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, Piece ID: DPLALMJC4JPC5D7BOK5SOSDFDJFDDPGSJWP4XDDQE22AMFNAKATQ) context deadline exceeded ======LOGS====== gment. Inline: false Nov 16 13:42:52.666975419 (173h38m51.21844588s) [upload=152907]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 16 13:42:52.666980539 (173h38m51.21845099s) [upload=152907]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 16 13:42:53.024947049 (173h38m51.5764175s) [seg_pos={0 0}, upload=152907]: Begin upload segment... Nov 16 13:42:53.025007074 (173h38m51.576477535s) [seg_pos={0 0}, upload=152907]: Done begin upload segment. Nov 16 13:42:53.025087707 (173h38m51.576558158s) [noise=true , address=77.90.31.14:4662 , piece_id=S5PPMCKIKSOCLUJ4CEH7ZDW3TVUZ7GTFFEIPCQK6FX25UIOUQHHQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.025098377 (173h38m51.576568828s) [noise=true , address=145.79.227.31:4797 , piece_id=UKP3MHXBSALCWFIHCTSR5JFFXEWWAWGDS4JYUYS2JEREQQKTPJSA, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.025961451 (173h38m51.577431912s) [noise=true , address=45.91.116.6:4974 , piece_id=ZTS3H7E6A3VJECLRRKRKSVX5PH3VSH7F2HAUPQSPH56NCI6RTFBQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.026101888 (173h38m51.577572339s) [noise=true , address=145.79.227.10:4402 , piece_id=DNT5QVAMRO5OMBNVTDRRUKACMM7M5472LD5E4ASKYIBKBNWYAVRA, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.026397721 (173h38m51.577868182s) [noise=true , address=145.79.227.11:4732 , piece_id=GR6SASHNUTHAEOUPK7DWKVPHRGMG72FQMF4RINJ5NTIQQGFCVZZQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.026543650 (173h38m51.578014111s) [noise=true , address=145.79.227.20:4437 , piece_id=G5P6OYXVAAQDZKTPPHDNZLMYTT6L2TFWKH7N4F7L6742BWHOQGBQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.026887915 (173h38m51.578358366s) [noise=true , address=45.91.116.14:4189 , piece_id=NUMS6Q3F2OUAQ2PSKE2CZK2WKRQLZUZPWVBA3UA6UNKDQACPYSJQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.027039624 (173h38m51.578510085s) [noise=true , address=145.79.227.27:4691 , piece_id=NSX5S2D7ZSA5I57B4EAQEQXEVUMPKNMRKRV3RHWGWJ2DN7SWUPGQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.027374482 (173h38m51.578844973s) [noise=true , address=145.79.227.31:4798 , piece_id=DPLALMJC4JPC5D7BOK5SOSDFDJFDDPGSJWP4XDDQE22AMFNAKATQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.027472337 (173h38m51.578942798s) [noise=true , address=45.91.116.7:4010 , piece_id=EO7IEOCWEJAMZZU4BLIB5ALFACWTWZE3QCXU3FUY4FKC4P444ORA, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.027920821 (173h38m51.579391283s) [noise=true , address=145.79.227.23:4547 , piece_id=P4WX2ZDHV2Z6BFMF6IACIYW4DXKYFVSULUTF7EK2EMRO5OA5X5RQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.028194272 (173h38m51.579664743s) [noise=true , address=145.79.227.32:4834 , piece_id=CBK2UQ252EPFQDTOYS2X5WAUTJN4L2DTHGIOJXFZXWBOLGPRETXQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.028430262 (173h38m51.579900723s) [noise=true , address=145.79.227.9:4366 , piece_id=YFWF2YL2YPYVMP5DHJKRUGVUUS2ALG4DRJ3HUZCDZM4N3N676FNQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.029687726 (173h38m51.581158177s) [noise=true , address=145.79.227.20:4438 , piece_id=6HY2HDNJAZ552ZKXFJ7VNTQNBAQU5ZQLNKAG2NZZWZQHN6EEDG6Q, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.031784128 (173h38m51.583254589s) [noise=true , address=45.91.116.5:4298 , piece_id=ARGPAZE6CTMM6UXSMPSVHDBEQWKVGKY7WXOVLPB7UH3JZYMDB7JQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.034100830 (173h38m51.585571291s) [noise=true , address=77.90.31.10:4518 , piece_id=MSKQ66KE4WANAE4AWOD2IKY3PQN2YGZ5N4W3NXGKXSOLE7B6H6WQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:54.850200282 (173h38m53.401670733s) [upload=152907]: Next returned nil segment Nov 16 13:42:54.850219118 (173h38m53.401689569s) [upload=152907]: Waiting for error group managing segments... Nov 16 13:42:55.154333032 (173h38m53.705803493s) [noise=true , address=45.91.116.7:4010 , piece_id=EO7IEOCWEJAMZZU4BLIB5ALFACWTWZE3QCXU3FUY4FKC4P444ORA, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:42:55.201689040 (173h38m53.753159501s) [noise=true , address=77.90.31.10:4518 , piece_id=MSKQ66KE4WANAE4AWOD2IKY3PQN2YGZ5N4W3NXGKXSOLE7B6H6WQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:42:55.232313345 (173h38m53.783783816s) [noise=true , address=45.91.116.5:4298 , piece_id=ARGPAZE6CTMM6UXSMPSVHDBEQWKVGKY7WXOVLPB7UH3JZYMDB7JQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:42:55.260281393 (173h38m53.811751854s) [noise=true , address=45.91.116.6:4974 , piece_id=ZTS3H7E6A3VJECLRRKRKSVX5PH3VSH7F2HAUPQSPH56NCI6RTFBQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:00.229127085 (173h38m58.780597546s) [noise=true , address=145.79.227.20:4438 , piece_id=6HY2HDNJAZ552ZKXFJ7VNTQNBAQU5ZQLNKAG2NZZWZQHN6EEDG6Q, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:00.358241119 (173h38m58.909711581s) [noise=true , address=145.79.227.32:4834 , piece_id=CBK2UQ252EPFQDTOYS2X5WAUTJN4L2DTHGIOJXFZXWBOLGPRETXQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:00.646958215 (173h38m59.198428676s) [noise=true , address=145.79.227.31:4798 , piece_id=DPLALMJC4JPC5D7BOK5SOSDFDJFDDPGSJWP4XDDQE22AMFNAKATQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:00.908824080 (173h38m59.460294541s) [noise=true , address=145.79.227.23:4547 , piece_id=P4WX2ZDHV2Z6BFMF6IACIYW4DXKYFVSULUTF7EK2EMRO5OA5X5RQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:01.155346527 (173h38m59.706816989s) [noise=true , address=45.91.116.14:4189 , piece_id=NUMS6Q3F2OUAQ2PSKE2CZK2WKRQLZUZPWVBA3UA6UNKDQACPYSJQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:01.839444794 (173h39m0.390915255s) [noise=true , address=145.79.227.11:4732 , piece_id=GR6SASHNUTHAEOUPK7DWKVPHRGMG72FQMF4RINJ5NTIQQGFCVZZQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:02.591249579 (173h39m1.14272004s) [noise=true , address=145.79.227.10:4402 , piece_id=DNT5QVAMRO5OMBNVTDRRUKACMM7M5472LD5E4ASKYIBKBNWYAVRA, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:02.616126740 (173h39m1.167597202s) [noise=true , address=145.79.227.31:4797 , piece_id=UKP3MHXBSALCWFIHCTSR5JFFXEWWAWGDS4JYUYS2JEREQQKTPJSA, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:02.734605482 (173h39m1.286075943s) [noise=true , address=145.79.227.27:4691 , piece_id=NSX5S2D7ZSA5I57B4EAQEQXEVUMPKNMRKRV3RHWGWJ2DN7SWUPGQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:02.734638004 (173h39m1.286108475s) [seg_pos={0 0}, upload=152907]: Segment reached optimal threshold of 13 pieces. Nov 16 13:43:02.735049748 (173h39m1.286520209s) [noise=true , address=145.79.227.9:4366 , piece_id=YFWF2YL2YPYVMP5DHJKRUGVUUS2ALG4DRJ3HUZCDZM4N3N676FNQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy): protocol: write tcp 188.166.225.170:60100->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:60100->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60100->145.79.227.9:4366: use of closed network connection Nov 16 13:43:02.735060949 (173h39m1.28653141s) [noise=true , address=145.79.227.9:4366 , piece_id=YFWF2YL2YPYVMP5DHJKRUGVUUS2ALG4DRJ3HUZCDZM4N3N676FNQ, seg_pos={0 0}, upload=152907]: Long tail cancellation due to optimal threshold Nov 16 13:43:02.735509193 (173h39m1.286979654s) [noise=true , address=77.90.31.14:4662 , piece_id=S5PPMCKIKSOCLUJ4CEH7ZDW3TVUZ7GTFFEIPCQK6FX25UIOUQHHQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): protocol: write tcp 188.166.225.170:50196->77.90.31.14:4662: use of closed network connection; write tcp 188.166.225.170:50196->77.90.31.14:4662: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50196->77.90.31.14:4662: use of closed network connection Nov 16 13:43:02.735518792 (173h39m1.286989253s) [noise=true , address=77.90.31.14:4662 , piece_id=S5PPMCKIKSOCLUJ4CEH7ZDW3TVUZ7GTFFEIPCQK6FX25UIOUQHHQ, seg_pos={0 0}, upload=152907]: Long tail cancellation due to optimal threshold Nov 16 13:43:02.736224335 (173h39m1.287694796s) [noise=true , address=145.79.227.20:4437 , piece_id=G5P6OYXVAAQDZKTPPHDNZLMYTT6L2TFWKH7N4F7L6742BWHOQGBQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV): protocol: read tcp 188.166.225.170:35300->145.79.227.20:4437: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 13:43:02.736239203 (173h39m1.287709654s) [noise=true , address=145.79.227.20:4437 , piece_id=G5P6OYXVAAQDZKTPPHDNZLMYTT6L2TFWKH7N4F7L6742BWHOQGBQ, seg_pos={0 0}, upload=152907]: Long tail cancellation due to optimal threshold Nov 16 13:43:02.736270833 (173h39m1.287741294s) [seg_pos={0 0}, upload=152907]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 16 13:43:02.736437060 (173h39m1.287907521s) [upload=152907]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 16 13:43:02.736444273 (173h39m1.287914735s) [upload=152907]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 16 13:43:05.031396865 (173h39m3.582867336s) [upload=152907]: Done uploading segments... Nov 16 13:43:05.031422193 (173h39m3.582892654s) [upload=152907]: Upload finished. err: <nil> Duration: 7.944322055s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 9 out of 60 is failed (threshold=6) [..X......X..XX....X......................X........X..X.....X] duration: 5.002105324s trace: https://jaeger.dev.storj.io/trace/6d62f0753f4f7865 FAIL (2025-11-16 14:18:44 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.002131202s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 6.027912256s trace: https://jaeger.dev.storj.io/trace/3c77028f820d12f9 FAIL (2025-11-16 14:18:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, Piece ID: UKP3MHXBSALCWFIHCTSR5JFFXEWWAWGDS4JYUYS2JEREQQKTPJSA) context deadline exceeded; piecestore: (Node ID: 12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, Piece ID: DNT5QVAMRO5OMBNVTDRRUKACMM7M5472LD5E4ASKYIBKBNWYAVRA) context deadline exceeded; piecestore: (Node ID: 17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, Piece ID: DPLALMJC4JPC5D7BOK5SOSDFDJFDDPGSJWP4XDDQE22AMFNAKATQ) context deadline exceeded ======LOGS====== gment. Inline: false Nov 16 13:42:52.666975419 (173h38m51.21844588s) [upload=152907]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 16 13:42:52.666980539 (173h38m51.21845099s) [upload=152907]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 16 13:42:53.024947049 (173h38m51.5764175s) [seg_pos={0 0}, upload=152907]: Begin upload segment... Nov 16 13:42:53.025007074 (173h38m51.576477535s) [seg_pos={0 0}, upload=152907]: Done begin upload segment. Nov 16 13:42:53.025087707 (173h38m51.576558158s) [noise=true , address=77.90.31.14:4662 , piece_id=S5PPMCKIKSOCLUJ4CEH7ZDW3TVUZ7GTFFEIPCQK6FX25UIOUQHHQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.025098377 (173h38m51.576568828s) [noise=true , address=145.79.227.31:4797 , piece_id=UKP3MHXBSALCWFIHCTSR5JFFXEWWAWGDS4JYUYS2JEREQQKTPJSA, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.025961451 (173h38m51.577431912s) [noise=true , address=45.91.116.6:4974 , piece_id=ZTS3H7E6A3VJECLRRKRKSVX5PH3VSH7F2HAUPQSPH56NCI6RTFBQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.026101888 (173h38m51.577572339s) [noise=true , address=145.79.227.10:4402 , piece_id=DNT5QVAMRO5OMBNVTDRRUKACMM7M5472LD5E4ASKYIBKBNWYAVRA, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.026397721 (173h38m51.577868182s) [noise=true , address=145.79.227.11:4732 , piece_id=GR6SASHNUTHAEOUPK7DWKVPHRGMG72FQMF4RINJ5NTIQQGFCVZZQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.026543650 (173h38m51.578014111s) [noise=true , address=145.79.227.20:4437 , piece_id=G5P6OYXVAAQDZKTPPHDNZLMYTT6L2TFWKH7N4F7L6742BWHOQGBQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.026887915 (173h38m51.578358366s) [noise=true , address=45.91.116.14:4189 , piece_id=NUMS6Q3F2OUAQ2PSKE2CZK2WKRQLZUZPWVBA3UA6UNKDQACPYSJQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.027039624 (173h38m51.578510085s) [noise=true , address=145.79.227.27:4691 , piece_id=NSX5S2D7ZSA5I57B4EAQEQXEVUMPKNMRKRV3RHWGWJ2DN7SWUPGQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.027374482 (173h38m51.578844973s) [noise=true , address=145.79.227.31:4798 , piece_id=DPLALMJC4JPC5D7BOK5SOSDFDJFDDPGSJWP4XDDQE22AMFNAKATQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.027472337 (173h38m51.578942798s) [noise=true , address=45.91.116.7:4010 , piece_id=EO7IEOCWEJAMZZU4BLIB5ALFACWTWZE3QCXU3FUY4FKC4P444ORA, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.027920821 (173h38m51.579391283s) [noise=true , address=145.79.227.23:4547 , piece_id=P4WX2ZDHV2Z6BFMF6IACIYW4DXKYFVSULUTF7EK2EMRO5OA5X5RQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.028194272 (173h38m51.579664743s) [noise=true , address=145.79.227.32:4834 , piece_id=CBK2UQ252EPFQDTOYS2X5WAUTJN4L2DTHGIOJXFZXWBOLGPRETXQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.028430262 (173h38m51.579900723s) [noise=true , address=145.79.227.9:4366 , piece_id=YFWF2YL2YPYVMP5DHJKRUGVUUS2ALG4DRJ3HUZCDZM4N3N676FNQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.029687726 (173h38m51.581158177s) [noise=true , address=145.79.227.20:4438 , piece_id=6HY2HDNJAZ552ZKXFJ7VNTQNBAQU5ZQLNKAG2NZZWZQHN6EEDG6Q, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.031784128 (173h38m51.583254589s) [noise=true , address=45.91.116.5:4298 , piece_id=ARGPAZE6CTMM6UXSMPSVHDBEQWKVGKY7WXOVLPB7UH3JZYMDB7JQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.034100830 (173h38m51.585571291s) [noise=true , address=77.90.31.10:4518 , piece_id=MSKQ66KE4WANAE4AWOD2IKY3PQN2YGZ5N4W3NXGKXSOLE7B6H6WQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:54.850200282 (173h38m53.401670733s) [upload=152907]: Next returned nil segment Nov 16 13:42:54.850219118 (173h38m53.401689569s) [upload=152907]: Waiting for error group managing segments... Nov 16 13:42:55.154333032 (173h38m53.705803493s) [noise=true , address=45.91.116.7:4010 , piece_id=EO7IEOCWEJAMZZU4BLIB5ALFACWTWZE3QCXU3FUY4FKC4P444ORA, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:42:55.201689040 (173h38m53.753159501s) [noise=true , address=77.90.31.10:4518 , piece_id=MSKQ66KE4WANAE4AWOD2IKY3PQN2YGZ5N4W3NXGKXSOLE7B6H6WQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:42:55.232313345 (173h38m53.783783816s) [noise=true , address=45.91.116.5:4298 , piece_id=ARGPAZE6CTMM6UXSMPSVHDBEQWKVGKY7WXOVLPB7UH3JZYMDB7JQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:42:55.260281393 (173h38m53.811751854s) [noise=true , address=45.91.116.6:4974 , piece_id=ZTS3H7E6A3VJECLRRKRKSVX5PH3VSH7F2HAUPQSPH56NCI6RTFBQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:00.229127085 (173h38m58.780597546s) [noise=true , address=145.79.227.20:4438 , piece_id=6HY2HDNJAZ552ZKXFJ7VNTQNBAQU5ZQLNKAG2NZZWZQHN6EEDG6Q, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:00.358241119 (173h38m58.909711581s) [noise=true , address=145.79.227.32:4834 , piece_id=CBK2UQ252EPFQDTOYS2X5WAUTJN4L2DTHGIOJXFZXWBOLGPRETXQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:00.646958215 (173h38m59.198428676s) [noise=true , address=145.79.227.31:4798 , piece_id=DPLALMJC4JPC5D7BOK5SOSDFDJFDDPGSJWP4XDDQE22AMFNAKATQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:00.908824080 (173h38m59.460294541s) [noise=true , address=145.79.227.23:4547 , piece_id=P4WX2ZDHV2Z6BFMF6IACIYW4DXKYFVSULUTF7EK2EMRO5OA5X5RQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:01.155346527 (173h38m59.706816989s) [noise=true , address=45.91.116.14:4189 , piece_id=NUMS6Q3F2OUAQ2PSKE2CZK2WKRQLZUZPWVBA3UA6UNKDQACPYSJQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:01.839444794 (173h39m0.390915255s) [noise=true , address=145.79.227.11:4732 , piece_id=GR6SASHNUTHAEOUPK7DWKVPHRGMG72FQMF4RINJ5NTIQQGFCVZZQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:02.591249579 (173h39m1.14272004s) [noise=true , address=145.79.227.10:4402 , piece_id=DNT5QVAMRO5OMBNVTDRRUKACMM7M5472LD5E4ASKYIBKBNWYAVRA, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:02.616126740 (173h39m1.167597202s) [noise=true , address=145.79.227.31:4797 , piece_id=UKP3MHXBSALCWFIHCTSR5JFFXEWWAWGDS4JYUYS2JEREQQKTPJSA, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:02.734605482 (173h39m1.286075943s) [noise=true , address=145.79.227.27:4691 , piece_id=NSX5S2D7ZSA5I57B4EAQEQXEVUMPKNMRKRV3RHWGWJ2DN7SWUPGQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:02.734638004 (173h39m1.286108475s) [seg_pos={0 0}, upload=152907]: Segment reached optimal threshold of 13 pieces. Nov 16 13:43:02.735049748 (173h39m1.286520209s) [noise=true , address=145.79.227.9:4366 , piece_id=YFWF2YL2YPYVMP5DHJKRUGVUUS2ALG4DRJ3HUZCDZM4N3N676FNQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy): protocol: write tcp 188.166.225.170:60100->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:60100->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60100->145.79.227.9:4366: use of closed network connection Nov 16 13:43:02.735060949 (173h39m1.28653141s) [noise=true , address=145.79.227.9:4366 , piece_id=YFWF2YL2YPYVMP5DHJKRUGVUUS2ALG4DRJ3HUZCDZM4N3N676FNQ, seg_pos={0 0}, upload=152907]: Long tail cancellation due to optimal threshold Nov 16 13:43:02.735509193 (173h39m1.286979654s) [noise=true , address=77.90.31.14:4662 , piece_id=S5PPMCKIKSOCLUJ4CEH7ZDW3TVUZ7GTFFEIPCQK6FX25UIOUQHHQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): protocol: write tcp 188.166.225.170:50196->77.90.31.14:4662: use of closed network connection; write tcp 188.166.225.170:50196->77.90.31.14:4662: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50196->77.90.31.14:4662: use of closed network connection Nov 16 13:43:02.735518792 (173h39m1.286989253s) [noise=true , address=77.90.31.14:4662 , piece_id=S5PPMCKIKSOCLUJ4CEH7ZDW3TVUZ7GTFFEIPCQK6FX25UIOUQHHQ, seg_pos={0 0}, upload=152907]: Long tail cancellation due to optimal threshold Nov 16 13:43:02.736224335 (173h39m1.287694796s) [noise=true , address=145.79.227.20:4437 , piece_id=G5P6OYXVAAQDZKTPPHDNZLMYTT6L2TFWKH7N4F7L6742BWHOQGBQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV): protocol: read tcp 188.166.225.170:35300->145.79.227.20:4437: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 13:43:02.736239203 (173h39m1.287709654s) [noise=true , address=145.79.227.20:4437 , piece_id=G5P6OYXVAAQDZKTPPHDNZLMYTT6L2TFWKH7N4F7L6742BWHOQGBQ, seg_pos={0 0}, upload=152907]: Long tail cancellation due to optimal threshold Nov 16 13:43:02.736270833 (173h39m1.287741294s) [seg_pos={0 0}, upload=152907]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 16 13:43:02.736437060 (173h39m1.287907521s) [upload=152907]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 16 13:43:02.736444273 (173h39m1.287914735s) [upload=152907]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 16 13:43:05.031396865 (173h39m3.582867336s) [upload=152907]: Done uploading segments... Nov 16 13:43:05.031422193 (173h39m3.582892654s) [upload=152907]: Upload finished. err: <nil> Duration: 6.027941543s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 8 out of 60 is failed (threshold=6) [...X......X..XX....X......................X........X..X.....] duration: 3.503371061s trace: https://jaeger.dev.storj.io/trace/3dcf6f8d25fcd7d8 FAIL (2025-11-16 14:17:32 UTC) orbiter warning: context deadline exceeded Duration: 3.503437237s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 6.278215616s trace: https://jaeger.dev.storj.io/trace/3c77028f820d12f9 FAIL (2025-11-16 14:16:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, Piece ID: UKP3MHXBSALCWFIHCTSR5JFFXEWWAWGDS4JYUYS2JEREQQKTPJSA) context deadline exceeded; piecestore: (Node ID: 12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, Piece ID: DNT5QVAMRO5OMBNVTDRRUKACMM7M5472LD5E4ASKYIBKBNWYAVRA) context deadline exceeded; piecestore: (Node ID: 17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, Piece ID: DPLALMJC4JPC5D7BOK5SOSDFDJFDDPGSJWP4XDDQE22AMFNAKATQ) context deadline exceeded ======LOGS====== gment. Inline: false Nov 16 13:42:52.666975419 (173h38m51.21844588s) [upload=152907]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 16 13:42:52.666980539 (173h38m51.21845099s) [upload=152907]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 16 13:42:53.024947049 (173h38m51.5764175s) [seg_pos={0 0}, upload=152907]: Begin upload segment... Nov 16 13:42:53.025007074 (173h38m51.576477535s) [seg_pos={0 0}, upload=152907]: Done begin upload segment. Nov 16 13:42:53.025087707 (173h38m51.576558158s) [noise=true , address=77.90.31.14:4662 , piece_id=S5PPMCKIKSOCLUJ4CEH7ZDW3TVUZ7GTFFEIPCQK6FX25UIOUQHHQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.025098377 (173h38m51.576568828s) [noise=true , address=145.79.227.31:4797 , piece_id=UKP3MHXBSALCWFIHCTSR5JFFXEWWAWGDS4JYUYS2JEREQQKTPJSA, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.025961451 (173h38m51.577431912s) [noise=true , address=45.91.116.6:4974 , piece_id=ZTS3H7E6A3VJECLRRKRKSVX5PH3VSH7F2HAUPQSPH56NCI6RTFBQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.026101888 (173h38m51.577572339s) [noise=true , address=145.79.227.10:4402 , piece_id=DNT5QVAMRO5OMBNVTDRRUKACMM7M5472LD5E4ASKYIBKBNWYAVRA, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.026397721 (173h38m51.577868182s) [noise=true , address=145.79.227.11:4732 , piece_id=GR6SASHNUTHAEOUPK7DWKVPHRGMG72FQMF4RINJ5NTIQQGFCVZZQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.026543650 (173h38m51.578014111s) [noise=true , address=145.79.227.20:4437 , piece_id=G5P6OYXVAAQDZKTPPHDNZLMYTT6L2TFWKH7N4F7L6742BWHOQGBQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.026887915 (173h38m51.578358366s) [noise=true , address=45.91.116.14:4189 , piece_id=NUMS6Q3F2OUAQ2PSKE2CZK2WKRQLZUZPWVBA3UA6UNKDQACPYSJQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.027039624 (173h38m51.578510085s) [noise=true , address=145.79.227.27:4691 , piece_id=NSX5S2D7ZSA5I57B4EAQEQXEVUMPKNMRKRV3RHWGWJ2DN7SWUPGQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.027374482 (173h38m51.578844973s) [noise=true , address=145.79.227.31:4798 , piece_id=DPLALMJC4JPC5D7BOK5SOSDFDJFDDPGSJWP4XDDQE22AMFNAKATQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.027472337 (173h38m51.578942798s) [noise=true , address=45.91.116.7:4010 , piece_id=EO7IEOCWEJAMZZU4BLIB5ALFACWTWZE3QCXU3FUY4FKC4P444ORA, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.027920821 (173h38m51.579391283s) [noise=true , address=145.79.227.23:4547 , piece_id=P4WX2ZDHV2Z6BFMF6IACIYW4DXKYFVSULUTF7EK2EMRO5OA5X5RQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.028194272 (173h38m51.579664743s) [noise=true , address=145.79.227.32:4834 , piece_id=CBK2UQ252EPFQDTOYS2X5WAUTJN4L2DTHGIOJXFZXWBOLGPRETXQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.028430262 (173h38m51.579900723s) [noise=true , address=145.79.227.9:4366 , piece_id=YFWF2YL2YPYVMP5DHJKRUGVUUS2ALG4DRJ3HUZCDZM4N3N676FNQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.029687726 (173h38m51.581158177s) [noise=true , address=145.79.227.20:4438 , piece_id=6HY2HDNJAZ552ZKXFJ7VNTQNBAQU5ZQLNKAG2NZZWZQHN6EEDG6Q, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.031784128 (173h38m51.583254589s) [noise=true , address=45.91.116.5:4298 , piece_id=ARGPAZE6CTMM6UXSMPSVHDBEQWKVGKY7WXOVLPB7UH3JZYMDB7JQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.034100830 (173h38m51.585571291s) [noise=true , address=77.90.31.10:4518 , piece_id=MSKQ66KE4WANAE4AWOD2IKY3PQN2YGZ5N4W3NXGKXSOLE7B6H6WQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:54.850200282 (173h38m53.401670733s) [upload=152907]: Next returned nil segment Nov 16 13:42:54.850219118 (173h38m53.401689569s) [upload=152907]: Waiting for error group managing segments... Nov 16 13:42:55.154333032 (173h38m53.705803493s) [noise=true , address=45.91.116.7:4010 , piece_id=EO7IEOCWEJAMZZU4BLIB5ALFACWTWZE3QCXU3FUY4FKC4P444ORA, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:42:55.201689040 (173h38m53.753159501s) [noise=true , address=77.90.31.10:4518 , piece_id=MSKQ66KE4WANAE4AWOD2IKY3PQN2YGZ5N4W3NXGKXSOLE7B6H6WQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:42:55.232313345 (173h38m53.783783816s) [noise=true , address=45.91.116.5:4298 , piece_id=ARGPAZE6CTMM6UXSMPSVHDBEQWKVGKY7WXOVLPB7UH3JZYMDB7JQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:42:55.260281393 (173h38m53.811751854s) [noise=true , address=45.91.116.6:4974 , piece_id=ZTS3H7E6A3VJECLRRKRKSVX5PH3VSH7F2HAUPQSPH56NCI6RTFBQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:00.229127085 (173h38m58.780597546s) [noise=true , address=145.79.227.20:4438 , piece_id=6HY2HDNJAZ552ZKXFJ7VNTQNBAQU5ZQLNKAG2NZZWZQHN6EEDG6Q, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:00.358241119 (173h38m58.909711581s) [noise=true , address=145.79.227.32:4834 , piece_id=CBK2UQ252EPFQDTOYS2X5WAUTJN4L2DTHGIOJXFZXWBOLGPRETXQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:00.646958215 (173h38m59.198428676s) [noise=true , address=145.79.227.31:4798 , piece_id=DPLALMJC4JPC5D7BOK5SOSDFDJFDDPGSJWP4XDDQE22AMFNAKATQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:00.908824080 (173h38m59.460294541s) [noise=true , address=145.79.227.23:4547 , piece_id=P4WX2ZDHV2Z6BFMF6IACIYW4DXKYFVSULUTF7EK2EMRO5OA5X5RQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:01.155346527 (173h38m59.706816989s) [noise=true , address=45.91.116.14:4189 , piece_id=NUMS6Q3F2OUAQ2PSKE2CZK2WKRQLZUZPWVBA3UA6UNKDQACPYSJQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:01.839444794 (173h39m0.390915255s) [noise=true , address=145.79.227.11:4732 , piece_id=GR6SASHNUTHAEOUPK7DWKVPHRGMG72FQMF4RINJ5NTIQQGFCVZZQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:02.591249579 (173h39m1.14272004s) [noise=true , address=145.79.227.10:4402 , piece_id=DNT5QVAMRO5OMBNVTDRRUKACMM7M5472LD5E4ASKYIBKBNWYAVRA, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:02.616126740 (173h39m1.167597202s) [noise=true , address=145.79.227.31:4797 , piece_id=UKP3MHXBSALCWFIHCTSR5JFFXEWWAWGDS4JYUYS2JEREQQKTPJSA, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:02.734605482 (173h39m1.286075943s) [noise=true , address=145.79.227.27:4691 , piece_id=NSX5S2D7ZSA5I57B4EAQEQXEVUMPKNMRKRV3RHWGWJ2DN7SWUPGQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:02.734638004 (173h39m1.286108475s) [seg_pos={0 0}, upload=152907]: Segment reached optimal threshold of 13 pieces. Nov 16 13:43:02.735049748 (173h39m1.286520209s) [noise=true , address=145.79.227.9:4366 , piece_id=YFWF2YL2YPYVMP5DHJKRUGVUUS2ALG4DRJ3HUZCDZM4N3N676FNQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy): protocol: write tcp 188.166.225.170:60100->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:60100->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60100->145.79.227.9:4366: use of closed network connection Nov 16 13:43:02.735060949 (173h39m1.28653141s) [noise=true , address=145.79.227.9:4366 , piece_id=YFWF2YL2YPYVMP5DHJKRUGVUUS2ALG4DRJ3HUZCDZM4N3N676FNQ, seg_pos={0 0}, upload=152907]: Long tail cancellation due to optimal threshold Nov 16 13:43:02.735509193 (173h39m1.286979654s) [noise=true , address=77.90.31.14:4662 , piece_id=S5PPMCKIKSOCLUJ4CEH7ZDW3TVUZ7GTFFEIPCQK6FX25UIOUQHHQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): protocol: write tcp 188.166.225.170:50196->77.90.31.14:4662: use of closed network connection; write tcp 188.166.225.170:50196->77.90.31.14:4662: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50196->77.90.31.14:4662: use of closed network connection Nov 16 13:43:02.735518792 (173h39m1.286989253s) [noise=true , address=77.90.31.14:4662 , piece_id=S5PPMCKIKSOCLUJ4CEH7ZDW3TVUZ7GTFFEIPCQK6FX25UIOUQHHQ, seg_pos={0 0}, upload=152907]: Long tail cancellation due to optimal threshold Nov 16 13:43:02.736224335 (173h39m1.287694796s) [noise=true , address=145.79.227.20:4437 , piece_id=G5P6OYXVAAQDZKTPPHDNZLMYTT6L2TFWKH7N4F7L6742BWHOQGBQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV): protocol: read tcp 188.166.225.170:35300->145.79.227.20:4437: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 13:43:02.736239203 (173h39m1.287709654s) [noise=true , address=145.79.227.20:4437 , piece_id=G5P6OYXVAAQDZKTPPHDNZLMYTT6L2TFWKH7N4F7L6742BWHOQGBQ, seg_pos={0 0}, upload=152907]: Long tail cancellation due to optimal threshold Nov 16 13:43:02.736270833 (173h39m1.287741294s) [seg_pos={0 0}, upload=152907]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 16 13:43:02.736437060 (173h39m1.287907521s) [upload=152907]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 16 13:43:02.736444273 (173h39m1.287914735s) [upload=152907]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 16 13:43:05.031396865 (173h39m3.582867336s) [upload=152907]: Done uploading segments... Nov 16 13:43:05.031422193 (173h39m3.582892654s) [upload=152907]: Upload finished. err: <nil> Duration: 6.278252466s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 8 out of 60 is failed (threshold=6) [....X......X..XX....X......................X........X..X....] duration: 3.486981887s trace: https://jaeger.dev.storj.io/trace/3dcf6f8d25fcd7d8 FAIL (2025-11-16 14:16:50 UTC) orbiter warning: context deadline exceeded Duration: 3.487082929s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 9 out of 60 is failed (threshold=6) [X....X......X..XX....X......................X........X..X...] duration: 3.508700686s trace: https://jaeger.dev.storj.io/trace/3dcf6f8d25fcd7d8 FAIL (2025-11-16 14:15:47 UTC) orbiter warning: context deadline exceeded Duration: 3.508725582s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 6.317870227s trace: https://jaeger.dev.storj.io/trace/3c77028f820d12f9 FAIL (2025-11-16 14:15:40 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, Piece ID: UKP3MHXBSALCWFIHCTSR5JFFXEWWAWGDS4JYUYS2JEREQQKTPJSA) context deadline exceeded; piecestore: (Node ID: 12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, Piece ID: DNT5QVAMRO5OMBNVTDRRUKACMM7M5472LD5E4ASKYIBKBNWYAVRA) context deadline exceeded; piecestore: (Node ID: 17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, Piece ID: DPLALMJC4JPC5D7BOK5SOSDFDJFDDPGSJWP4XDDQE22AMFNAKATQ) context deadline exceeded ======LOGS====== gment. Inline: false Nov 16 13:42:52.666975419 (173h38m51.21844588s) [upload=152907]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 16 13:42:52.666980539 (173h38m51.21845099s) [upload=152907]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 16 13:42:53.024947049 (173h38m51.5764175s) [seg_pos={0 0}, upload=152907]: Begin upload segment... Nov 16 13:42:53.025007074 (173h38m51.576477535s) [seg_pos={0 0}, upload=152907]: Done begin upload segment. Nov 16 13:42:53.025087707 (173h38m51.576558158s) [noise=true , address=77.90.31.14:4662 , piece_id=S5PPMCKIKSOCLUJ4CEH7ZDW3TVUZ7GTFFEIPCQK6FX25UIOUQHHQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.025098377 (173h38m51.576568828s) [noise=true , address=145.79.227.31:4797 , piece_id=UKP3MHXBSALCWFIHCTSR5JFFXEWWAWGDS4JYUYS2JEREQQKTPJSA, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.025961451 (173h38m51.577431912s) [noise=true , address=45.91.116.6:4974 , piece_id=ZTS3H7E6A3VJECLRRKRKSVX5PH3VSH7F2HAUPQSPH56NCI6RTFBQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.026101888 (173h38m51.577572339s) [noise=true , address=145.79.227.10:4402 , piece_id=DNT5QVAMRO5OMBNVTDRRUKACMM7M5472LD5E4ASKYIBKBNWYAVRA, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.026397721 (173h38m51.577868182s) [noise=true , address=145.79.227.11:4732 , piece_id=GR6SASHNUTHAEOUPK7DWKVPHRGMG72FQMF4RINJ5NTIQQGFCVZZQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.026543650 (173h38m51.578014111s) [noise=true , address=145.79.227.20:4437 , piece_id=G5P6OYXVAAQDZKTPPHDNZLMYTT6L2TFWKH7N4F7L6742BWHOQGBQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.026887915 (173h38m51.578358366s) [noise=true , address=45.91.116.14:4189 , piece_id=NUMS6Q3F2OUAQ2PSKE2CZK2WKRQLZUZPWVBA3UA6UNKDQACPYSJQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.027039624 (173h38m51.578510085s) [noise=true , address=145.79.227.27:4691 , piece_id=NSX5S2D7ZSA5I57B4EAQEQXEVUMPKNMRKRV3RHWGWJ2DN7SWUPGQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.027374482 (173h38m51.578844973s) [noise=true , address=145.79.227.31:4798 , piece_id=DPLALMJC4JPC5D7BOK5SOSDFDJFDDPGSJWP4XDDQE22AMFNAKATQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.027472337 (173h38m51.578942798s) [noise=true , address=45.91.116.7:4010 , piece_id=EO7IEOCWEJAMZZU4BLIB5ALFACWTWZE3QCXU3FUY4FKC4P444ORA, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.027920821 (173h38m51.579391283s) [noise=true , address=145.79.227.23:4547 , piece_id=P4WX2ZDHV2Z6BFMF6IACIYW4DXKYFVSULUTF7EK2EMRO5OA5X5RQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.028194272 (173h38m51.579664743s) [noise=true , address=145.79.227.32:4834 , piece_id=CBK2UQ252EPFQDTOYS2X5WAUTJN4L2DTHGIOJXFZXWBOLGPRETXQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.028430262 (173h38m51.579900723s) [noise=true , address=145.79.227.9:4366 , piece_id=YFWF2YL2YPYVMP5DHJKRUGVUUS2ALG4DRJ3HUZCDZM4N3N676FNQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.029687726 (173h38m51.581158177s) [noise=true , address=145.79.227.20:4438 , piece_id=6HY2HDNJAZ552ZKXFJ7VNTQNBAQU5ZQLNKAG2NZZWZQHN6EEDG6Q, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.031784128 (173h38m51.583254589s) [noise=true , address=45.91.116.5:4298 , piece_id=ARGPAZE6CTMM6UXSMPSVHDBEQWKVGKY7WXOVLPB7UH3JZYMDB7JQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.034100830 (173h38m51.585571291s) [noise=true , address=77.90.31.10:4518 , piece_id=MSKQ66KE4WANAE4AWOD2IKY3PQN2YGZ5N4W3NXGKXSOLE7B6H6WQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:54.850200282 (173h38m53.401670733s) [upload=152907]: Next returned nil segment Nov 16 13:42:54.850219118 (173h38m53.401689569s) [upload=152907]: Waiting for error group managing segments... Nov 16 13:42:55.154333032 (173h38m53.705803493s) [noise=true , address=45.91.116.7:4010 , piece_id=EO7IEOCWEJAMZZU4BLIB5ALFACWTWZE3QCXU3FUY4FKC4P444ORA, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:42:55.201689040 (173h38m53.753159501s) [noise=true , address=77.90.31.10:4518 , piece_id=MSKQ66KE4WANAE4AWOD2IKY3PQN2YGZ5N4W3NXGKXSOLE7B6H6WQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:42:55.232313345 (173h38m53.783783816s) [noise=true , address=45.91.116.5:4298 , piece_id=ARGPAZE6CTMM6UXSMPSVHDBEQWKVGKY7WXOVLPB7UH3JZYMDB7JQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:42:55.260281393 (173h38m53.811751854s) [noise=true , address=45.91.116.6:4974 , piece_id=ZTS3H7E6A3VJECLRRKRKSVX5PH3VSH7F2HAUPQSPH56NCI6RTFBQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:00.229127085 (173h38m58.780597546s) [noise=true , address=145.79.227.20:4438 , piece_id=6HY2HDNJAZ552ZKXFJ7VNTQNBAQU5ZQLNKAG2NZZWZQHN6EEDG6Q, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:00.358241119 (173h38m58.909711581s) [noise=true , address=145.79.227.32:4834 , piece_id=CBK2UQ252EPFQDTOYS2X5WAUTJN4L2DTHGIOJXFZXWBOLGPRETXQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:00.646958215 (173h38m59.198428676s) [noise=true , address=145.79.227.31:4798 , piece_id=DPLALMJC4JPC5D7BOK5SOSDFDJFDDPGSJWP4XDDQE22AMFNAKATQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:00.908824080 (173h38m59.460294541s) [noise=true , address=145.79.227.23:4547 , piece_id=P4WX2ZDHV2Z6BFMF6IACIYW4DXKYFVSULUTF7EK2EMRO5OA5X5RQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:01.155346527 (173h38m59.706816989s) [noise=true , address=45.91.116.14:4189 , piece_id=NUMS6Q3F2OUAQ2PSKE2CZK2WKRQLZUZPWVBA3UA6UNKDQACPYSJQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:01.839444794 (173h39m0.390915255s) [noise=true , address=145.79.227.11:4732 , piece_id=GR6SASHNUTHAEOUPK7DWKVPHRGMG72FQMF4RINJ5NTIQQGFCVZZQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:02.591249579 (173h39m1.14272004s) [noise=true , address=145.79.227.10:4402 , piece_id=DNT5QVAMRO5OMBNVTDRRUKACMM7M5472LD5E4ASKYIBKBNWYAVRA, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:02.616126740 (173h39m1.167597202s) [noise=true , address=145.79.227.31:4797 , piece_id=UKP3MHXBSALCWFIHCTSR5JFFXEWWAWGDS4JYUYS2JEREQQKTPJSA, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:02.734605482 (173h39m1.286075943s) [noise=true , address=145.79.227.27:4691 , piece_id=NSX5S2D7ZSA5I57B4EAQEQXEVUMPKNMRKRV3RHWGWJ2DN7SWUPGQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:02.734638004 (173h39m1.286108475s) [seg_pos={0 0}, upload=152907]: Segment reached optimal threshold of 13 pieces. Nov 16 13:43:02.735049748 (173h39m1.286520209s) [noise=true , address=145.79.227.9:4366 , piece_id=YFWF2YL2YPYVMP5DHJKRUGVUUS2ALG4DRJ3HUZCDZM4N3N676FNQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy): protocol: write tcp 188.166.225.170:60100->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:60100->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60100->145.79.227.9:4366: use of closed network connection Nov 16 13:43:02.735060949 (173h39m1.28653141s) [noise=true , address=145.79.227.9:4366 , piece_id=YFWF2YL2YPYVMP5DHJKRUGVUUS2ALG4DRJ3HUZCDZM4N3N676FNQ, seg_pos={0 0}, upload=152907]: Long tail cancellation due to optimal threshold Nov 16 13:43:02.735509193 (173h39m1.286979654s) [noise=true , address=77.90.31.14:4662 , piece_id=S5PPMCKIKSOCLUJ4CEH7ZDW3TVUZ7GTFFEIPCQK6FX25UIOUQHHQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): protocol: write tcp 188.166.225.170:50196->77.90.31.14:4662: use of closed network connection; write tcp 188.166.225.170:50196->77.90.31.14:4662: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50196->77.90.31.14:4662: use of closed network connection Nov 16 13:43:02.735518792 (173h39m1.286989253s) [noise=true , address=77.90.31.14:4662 , piece_id=S5PPMCKIKSOCLUJ4CEH7ZDW3TVUZ7GTFFEIPCQK6FX25UIOUQHHQ, seg_pos={0 0}, upload=152907]: Long tail cancellation due to optimal threshold Nov 16 13:43:02.736224335 (173h39m1.287694796s) [noise=true , address=145.79.227.20:4437 , piece_id=G5P6OYXVAAQDZKTPPHDNZLMYTT6L2TFWKH7N4F7L6742BWHOQGBQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV): protocol: read tcp 188.166.225.170:35300->145.79.227.20:4437: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 13:43:02.736239203 (173h39m1.287709654s) [noise=true , address=145.79.227.20:4437 , piece_id=G5P6OYXVAAQDZKTPPHDNZLMYTT6L2TFWKH7N4F7L6742BWHOQGBQ, seg_pos={0 0}, upload=152907]: Long tail cancellation due to optimal threshold Nov 16 13:43:02.736270833 (173h39m1.287741294s) [seg_pos={0 0}, upload=152907]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 16 13:43:02.736437060 (173h39m1.287907521s) [upload=152907]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 16 13:43:02.736444273 (173h39m1.287914735s) [upload=152907]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 16 13:43:05.031396865 (173h39m3.582867336s) [upload=152907]: Done uploading segments... Nov 16 13:43:05.031422193 (173h39m3.582892654s) [upload=152907]: Upload finished. err: <nil> Duration: 6.317899053s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 6.164531175s trace: https://jaeger.dev.storj.io/trace/3c77028f820d12f9 FAIL (2025-11-16 14:15:13 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, Piece ID: UKP3MHXBSALCWFIHCTSR5JFFXEWWAWGDS4JYUYS2JEREQQKTPJSA) context deadline exceeded; piecestore: (Node ID: 12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, Piece ID: DNT5QVAMRO5OMBNVTDRRUKACMM7M5472LD5E4ASKYIBKBNWYAVRA) context deadline exceeded; piecestore: (Node ID: 17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, Piece ID: DPLALMJC4JPC5D7BOK5SOSDFDJFDDPGSJWP4XDDQE22AMFNAKATQ) context deadline exceeded ======LOGS====== gment. Inline: false Nov 16 13:42:52.666975419 (173h38m51.21844588s) [upload=152907]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 16 13:42:52.666980539 (173h38m51.21845099s) [upload=152907]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 16 13:42:53.024947049 (173h38m51.5764175s) [seg_pos={0 0}, upload=152907]: Begin upload segment... Nov 16 13:42:53.025007074 (173h38m51.576477535s) [seg_pos={0 0}, upload=152907]: Done begin upload segment. Nov 16 13:42:53.025087707 (173h38m51.576558158s) [noise=true , address=77.90.31.14:4662 , piece_id=S5PPMCKIKSOCLUJ4CEH7ZDW3TVUZ7GTFFEIPCQK6FX25UIOUQHHQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.025098377 (173h38m51.576568828s) [noise=true , address=145.79.227.31:4797 , piece_id=UKP3MHXBSALCWFIHCTSR5JFFXEWWAWGDS4JYUYS2JEREQQKTPJSA, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.025961451 (173h38m51.577431912s) [noise=true , address=45.91.116.6:4974 , piece_id=ZTS3H7E6A3VJECLRRKRKSVX5PH3VSH7F2HAUPQSPH56NCI6RTFBQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.026101888 (173h38m51.577572339s) [noise=true , address=145.79.227.10:4402 , piece_id=DNT5QVAMRO5OMBNVTDRRUKACMM7M5472LD5E4ASKYIBKBNWYAVRA, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.026397721 (173h38m51.577868182s) [noise=true , address=145.79.227.11:4732 , piece_id=GR6SASHNUTHAEOUPK7DWKVPHRGMG72FQMF4RINJ5NTIQQGFCVZZQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.026543650 (173h38m51.578014111s) [noise=true , address=145.79.227.20:4437 , piece_id=G5P6OYXVAAQDZKTPPHDNZLMYTT6L2TFWKH7N4F7L6742BWHOQGBQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.026887915 (173h38m51.578358366s) [noise=true , address=45.91.116.14:4189 , piece_id=NUMS6Q3F2OUAQ2PSKE2CZK2WKRQLZUZPWVBA3UA6UNKDQACPYSJQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.027039624 (173h38m51.578510085s) [noise=true , address=145.79.227.27:4691 , piece_id=NSX5S2D7ZSA5I57B4EAQEQXEVUMPKNMRKRV3RHWGWJ2DN7SWUPGQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.027374482 (173h38m51.578844973s) [noise=true , address=145.79.227.31:4798 , piece_id=DPLALMJC4JPC5D7BOK5SOSDFDJFDDPGSJWP4XDDQE22AMFNAKATQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.027472337 (173h38m51.578942798s) [noise=true , address=45.91.116.7:4010 , piece_id=EO7IEOCWEJAMZZU4BLIB5ALFACWTWZE3QCXU3FUY4FKC4P444ORA, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.027920821 (173h38m51.579391283s) [noise=true , address=145.79.227.23:4547 , piece_id=P4WX2ZDHV2Z6BFMF6IACIYW4DXKYFVSULUTF7EK2EMRO5OA5X5RQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.028194272 (173h38m51.579664743s) [noise=true , address=145.79.227.32:4834 , piece_id=CBK2UQ252EPFQDTOYS2X5WAUTJN4L2DTHGIOJXFZXWBOLGPRETXQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.028430262 (173h38m51.579900723s) [noise=true , address=145.79.227.9:4366 , piece_id=YFWF2YL2YPYVMP5DHJKRUGVUUS2ALG4DRJ3HUZCDZM4N3N676FNQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.029687726 (173h38m51.581158177s) [noise=true , address=145.79.227.20:4438 , piece_id=6HY2HDNJAZ552ZKXFJ7VNTQNBAQU5ZQLNKAG2NZZWZQHN6EEDG6Q, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.031784128 (173h38m51.583254589s) [noise=true , address=45.91.116.5:4298 , piece_id=ARGPAZE6CTMM6UXSMPSVHDBEQWKVGKY7WXOVLPB7UH3JZYMDB7JQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.034100830 (173h38m51.585571291s) [noise=true , address=77.90.31.10:4518 , piece_id=MSKQ66KE4WANAE4AWOD2IKY3PQN2YGZ5N4W3NXGKXSOLE7B6H6WQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:54.850200282 (173h38m53.401670733s) [upload=152907]: Next returned nil segment Nov 16 13:42:54.850219118 (173h38m53.401689569s) [upload=152907]: Waiting for error group managing segments... Nov 16 13:42:55.154333032 (173h38m53.705803493s) [noise=true , address=45.91.116.7:4010 , piece_id=EO7IEOCWEJAMZZU4BLIB5ALFACWTWZE3QCXU3FUY4FKC4P444ORA, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:42:55.201689040 (173h38m53.753159501s) [noise=true , address=77.90.31.10:4518 , piece_id=MSKQ66KE4WANAE4AWOD2IKY3PQN2YGZ5N4W3NXGKXSOLE7B6H6WQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:42:55.232313345 (173h38m53.783783816s) [noise=true , address=45.91.116.5:4298 , piece_id=ARGPAZE6CTMM6UXSMPSVHDBEQWKVGKY7WXOVLPB7UH3JZYMDB7JQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:42:55.260281393 (173h38m53.811751854s) [noise=true , address=45.91.116.6:4974 , piece_id=ZTS3H7E6A3VJECLRRKRKSVX5PH3VSH7F2HAUPQSPH56NCI6RTFBQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:00.229127085 (173h38m58.780597546s) [noise=true , address=145.79.227.20:4438 , piece_id=6HY2HDNJAZ552ZKXFJ7VNTQNBAQU5ZQLNKAG2NZZWZQHN6EEDG6Q, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:00.358241119 (173h38m58.909711581s) [noise=true , address=145.79.227.32:4834 , piece_id=CBK2UQ252EPFQDTOYS2X5WAUTJN4L2DTHGIOJXFZXWBOLGPRETXQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:00.646958215 (173h38m59.198428676s) [noise=true , address=145.79.227.31:4798 , piece_id=DPLALMJC4JPC5D7BOK5SOSDFDJFDDPGSJWP4XDDQE22AMFNAKATQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:00.908824080 (173h38m59.460294541s) [noise=true , address=145.79.227.23:4547 , piece_id=P4WX2ZDHV2Z6BFMF6IACIYW4DXKYFVSULUTF7EK2EMRO5OA5X5RQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:01.155346527 (173h38m59.706816989s) [noise=true , address=45.91.116.14:4189 , piece_id=NUMS6Q3F2OUAQ2PSKE2CZK2WKRQLZUZPWVBA3UA6UNKDQACPYSJQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:01.839444794 (173h39m0.390915255s) [noise=true , address=145.79.227.11:4732 , piece_id=GR6SASHNUTHAEOUPK7DWKVPHRGMG72FQMF4RINJ5NTIQQGFCVZZQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:02.591249579 (173h39m1.14272004s) [noise=true , address=145.79.227.10:4402 , piece_id=DNT5QVAMRO5OMBNVTDRRUKACMM7M5472LD5E4ASKYIBKBNWYAVRA, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:02.616126740 (173h39m1.167597202s) [noise=true , address=145.79.227.31:4797 , piece_id=UKP3MHXBSALCWFIHCTSR5JFFXEWWAWGDS4JYUYS2JEREQQKTPJSA, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:02.734605482 (173h39m1.286075943s) [noise=true , address=145.79.227.27:4691 , piece_id=NSX5S2D7ZSA5I57B4EAQEQXEVUMPKNMRKRV3RHWGWJ2DN7SWUPGQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:02.734638004 (173h39m1.286108475s) [seg_pos={0 0}, upload=152907]: Segment reached optimal threshold of 13 pieces. Nov 16 13:43:02.735049748 (173h39m1.286520209s) [noise=true , address=145.79.227.9:4366 , piece_id=YFWF2YL2YPYVMP5DHJKRUGVUUS2ALG4DRJ3HUZCDZM4N3N676FNQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy): protocol: write tcp 188.166.225.170:60100->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:60100->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60100->145.79.227.9:4366: use of closed network connection Nov 16 13:43:02.735060949 (173h39m1.28653141s) [noise=true , address=145.79.227.9:4366 , piece_id=YFWF2YL2YPYVMP5DHJKRUGVUUS2ALG4DRJ3HUZCDZM4N3N676FNQ, seg_pos={0 0}, upload=152907]: Long tail cancellation due to optimal threshold Nov 16 13:43:02.735509193 (173h39m1.286979654s) [noise=true , address=77.90.31.14:4662 , piece_id=S5PPMCKIKSOCLUJ4CEH7ZDW3TVUZ7GTFFEIPCQK6FX25UIOUQHHQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): protocol: write tcp 188.166.225.170:50196->77.90.31.14:4662: use of closed network connection; write tcp 188.166.225.170:50196->77.90.31.14:4662: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50196->77.90.31.14:4662: use of closed network connection Nov 16 13:43:02.735518792 (173h39m1.286989253s) [noise=true , address=77.90.31.14:4662 , piece_id=S5PPMCKIKSOCLUJ4CEH7ZDW3TVUZ7GTFFEIPCQK6FX25UIOUQHHQ, seg_pos={0 0}, upload=152907]: Long tail cancellation due to optimal threshold Nov 16 13:43:02.736224335 (173h39m1.287694796s) [noise=true , address=145.79.227.20:4437 , piece_id=G5P6OYXVAAQDZKTPPHDNZLMYTT6L2TFWKH7N4F7L6742BWHOQGBQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV): protocol: read tcp 188.166.225.170:35300->145.79.227.20:4437: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 13:43:02.736239203 (173h39m1.287709654s) [noise=true , address=145.79.227.20:4437 , piece_id=G5P6OYXVAAQDZKTPPHDNZLMYTT6L2TFWKH7N4F7L6742BWHOQGBQ, seg_pos={0 0}, upload=152907]: Long tail cancellation due to optimal threshold Nov 16 13:43:02.736270833 (173h39m1.287741294s) [seg_pos={0 0}, upload=152907]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 16 13:43:02.736437060 (173h39m1.287907521s) [upload=152907]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 16 13:43:02.736444273 (173h39m1.287914735s) [upload=152907]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 16 13:43:05.031396865 (173h39m3.582867336s) [upload=152907]: Done uploading segments... Nov 16 13:43:05.031422193 (173h39m3.582892654s) [upload=152907]: Upload finished. err: <nil> Duration: 6.164575459s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 10 out of 60 is failed (threshold=6) [XX....X......X..XX....X......................X........X..X..] duration: 3.965759469s trace: https://jaeger.dev.storj.io/trace/3dcf6f8d25fcd7d8 FAIL (2025-11-16 14:14:57 UTC) orbiter warning: context deadline exceeded Duration: 3.965789446s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 6.065264168s trace: https://jaeger.dev.storj.io/trace/3c77028f820d12f9 FAIL (2025-11-16 14:14:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, Piece ID: UKP3MHXBSALCWFIHCTSR5JFFXEWWAWGDS4JYUYS2JEREQQKTPJSA) context deadline exceeded; piecestore: (Node ID: 12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, Piece ID: DNT5QVAMRO5OMBNVTDRRUKACMM7M5472LD5E4ASKYIBKBNWYAVRA) context deadline exceeded; piecestore: (Node ID: 17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, Piece ID: DPLALMJC4JPC5D7BOK5SOSDFDJFDDPGSJWP4XDDQE22AMFNAKATQ) context deadline exceeded ======LOGS====== gment. Inline: false Nov 16 13:42:52.666975419 (173h38m51.21844588s) [upload=152907]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 16 13:42:52.666980539 (173h38m51.21845099s) [upload=152907]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 16 13:42:53.024947049 (173h38m51.5764175s) [seg_pos={0 0}, upload=152907]: Begin upload segment... Nov 16 13:42:53.025007074 (173h38m51.576477535s) [seg_pos={0 0}, upload=152907]: Done begin upload segment. Nov 16 13:42:53.025087707 (173h38m51.576558158s) [noise=true , address=77.90.31.14:4662 , piece_id=S5PPMCKIKSOCLUJ4CEH7ZDW3TVUZ7GTFFEIPCQK6FX25UIOUQHHQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.025098377 (173h38m51.576568828s) [noise=true , address=145.79.227.31:4797 , piece_id=UKP3MHXBSALCWFIHCTSR5JFFXEWWAWGDS4JYUYS2JEREQQKTPJSA, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.025961451 (173h38m51.577431912s) [noise=true , address=45.91.116.6:4974 , piece_id=ZTS3H7E6A3VJECLRRKRKSVX5PH3VSH7F2HAUPQSPH56NCI6RTFBQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.026101888 (173h38m51.577572339s) [noise=true , address=145.79.227.10:4402 , piece_id=DNT5QVAMRO5OMBNVTDRRUKACMM7M5472LD5E4ASKYIBKBNWYAVRA, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.026397721 (173h38m51.577868182s) [noise=true , address=145.79.227.11:4732 , piece_id=GR6SASHNUTHAEOUPK7DWKVPHRGMG72FQMF4RINJ5NTIQQGFCVZZQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.026543650 (173h38m51.578014111s) [noise=true , address=145.79.227.20:4437 , piece_id=G5P6OYXVAAQDZKTPPHDNZLMYTT6L2TFWKH7N4F7L6742BWHOQGBQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.026887915 (173h38m51.578358366s) [noise=true , address=45.91.116.14:4189 , piece_id=NUMS6Q3F2OUAQ2PSKE2CZK2WKRQLZUZPWVBA3UA6UNKDQACPYSJQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.027039624 (173h38m51.578510085s) [noise=true , address=145.79.227.27:4691 , piece_id=NSX5S2D7ZSA5I57B4EAQEQXEVUMPKNMRKRV3RHWGWJ2DN7SWUPGQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.027374482 (173h38m51.578844973s) [noise=true , address=145.79.227.31:4798 , piece_id=DPLALMJC4JPC5D7BOK5SOSDFDJFDDPGSJWP4XDDQE22AMFNAKATQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.027472337 (173h38m51.578942798s) [noise=true , address=45.91.116.7:4010 , piece_id=EO7IEOCWEJAMZZU4BLIB5ALFACWTWZE3QCXU3FUY4FKC4P444ORA, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.027920821 (173h38m51.579391283s) [noise=true , address=145.79.227.23:4547 , piece_id=P4WX2ZDHV2Z6BFMF6IACIYW4DXKYFVSULUTF7EK2EMRO5OA5X5RQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.028194272 (173h38m51.579664743s) [noise=true , address=145.79.227.32:4834 , piece_id=CBK2UQ252EPFQDTOYS2X5WAUTJN4L2DTHGIOJXFZXWBOLGPRETXQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.028430262 (173h38m51.579900723s) [noise=true , address=145.79.227.9:4366 , piece_id=YFWF2YL2YPYVMP5DHJKRUGVUUS2ALG4DRJ3HUZCDZM4N3N676FNQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.029687726 (173h38m51.581158177s) [noise=true , address=145.79.227.20:4438 , piece_id=6HY2HDNJAZ552ZKXFJ7VNTQNBAQU5ZQLNKAG2NZZWZQHN6EEDG6Q, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.031784128 (173h38m51.583254589s) [noise=true , address=45.91.116.5:4298 , piece_id=ARGPAZE6CTMM6UXSMPSVHDBEQWKVGKY7WXOVLPB7UH3JZYMDB7JQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.034100830 (173h38m51.585571291s) [noise=true , address=77.90.31.10:4518 , piece_id=MSKQ66KE4WANAE4AWOD2IKY3PQN2YGZ5N4W3NXGKXSOLE7B6H6WQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:54.850200282 (173h38m53.401670733s) [upload=152907]: Next returned nil segment Nov 16 13:42:54.850219118 (173h38m53.401689569s) [upload=152907]: Waiting for error group managing segments... Nov 16 13:42:55.154333032 (173h38m53.705803493s) [noise=true , address=45.91.116.7:4010 , piece_id=EO7IEOCWEJAMZZU4BLIB5ALFACWTWZE3QCXU3FUY4FKC4P444ORA, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:42:55.201689040 (173h38m53.753159501s) [noise=true , address=77.90.31.10:4518 , piece_id=MSKQ66KE4WANAE4AWOD2IKY3PQN2YGZ5N4W3NXGKXSOLE7B6H6WQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:42:55.232313345 (173h38m53.783783816s) [noise=true , address=45.91.116.5:4298 , piece_id=ARGPAZE6CTMM6UXSMPSVHDBEQWKVGKY7WXOVLPB7UH3JZYMDB7JQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:42:55.260281393 (173h38m53.811751854s) [noise=true , address=45.91.116.6:4974 , piece_id=ZTS3H7E6A3VJECLRRKRKSVX5PH3VSH7F2HAUPQSPH56NCI6RTFBQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:00.229127085 (173h38m58.780597546s) [noise=true , address=145.79.227.20:4438 , piece_id=6HY2HDNJAZ552ZKXFJ7VNTQNBAQU5ZQLNKAG2NZZWZQHN6EEDG6Q, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:00.358241119 (173h38m58.909711581s) [noise=true , address=145.79.227.32:4834 , piece_id=CBK2UQ252EPFQDTOYS2X5WAUTJN4L2DTHGIOJXFZXWBOLGPRETXQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:00.646958215 (173h38m59.198428676s) [noise=true , address=145.79.227.31:4798 , piece_id=DPLALMJC4JPC5D7BOK5SOSDFDJFDDPGSJWP4XDDQE22AMFNAKATQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:00.908824080 (173h38m59.460294541s) [noise=true , address=145.79.227.23:4547 , piece_id=P4WX2ZDHV2Z6BFMF6IACIYW4DXKYFVSULUTF7EK2EMRO5OA5X5RQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:01.155346527 (173h38m59.706816989s) [noise=true , address=45.91.116.14:4189 , piece_id=NUMS6Q3F2OUAQ2PSKE2CZK2WKRQLZUZPWVBA3UA6UNKDQACPYSJQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:01.839444794 (173h39m0.390915255s) [noise=true , address=145.79.227.11:4732 , piece_id=GR6SASHNUTHAEOUPK7DWKVPHRGMG72FQMF4RINJ5NTIQQGFCVZZQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:02.591249579 (173h39m1.14272004s) [noise=true , address=145.79.227.10:4402 , piece_id=DNT5QVAMRO5OMBNVTDRRUKACMM7M5472LD5E4ASKYIBKBNWYAVRA, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:02.616126740 (173h39m1.167597202s) [noise=true , address=145.79.227.31:4797 , piece_id=UKP3MHXBSALCWFIHCTSR5JFFXEWWAWGDS4JYUYS2JEREQQKTPJSA, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:02.734605482 (173h39m1.286075943s) [noise=true , address=145.79.227.27:4691 , piece_id=NSX5S2D7ZSA5I57B4EAQEQXEVUMPKNMRKRV3RHWGWJ2DN7SWUPGQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:02.734638004 (173h39m1.286108475s) [seg_pos={0 0}, upload=152907]: Segment reached optimal threshold of 13 pieces. Nov 16 13:43:02.735049748 (173h39m1.286520209s) [noise=true , address=145.79.227.9:4366 , piece_id=YFWF2YL2YPYVMP5DHJKRUGVUUS2ALG4DRJ3HUZCDZM4N3N676FNQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy): protocol: write tcp 188.166.225.170:60100->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:60100->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60100->145.79.227.9:4366: use of closed network connection Nov 16 13:43:02.735060949 (173h39m1.28653141s) [noise=true , address=145.79.227.9:4366 , piece_id=YFWF2YL2YPYVMP5DHJKRUGVUUS2ALG4DRJ3HUZCDZM4N3N676FNQ, seg_pos={0 0}, upload=152907]: Long tail cancellation due to optimal threshold Nov 16 13:43:02.735509193 (173h39m1.286979654s) [noise=true , address=77.90.31.14:4662 , piece_id=S5PPMCKIKSOCLUJ4CEH7ZDW3TVUZ7GTFFEIPCQK6FX25UIOUQHHQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): protocol: write tcp 188.166.225.170:50196->77.90.31.14:4662: use of closed network connection; write tcp 188.166.225.170:50196->77.90.31.14:4662: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50196->77.90.31.14:4662: use of closed network connection Nov 16 13:43:02.735518792 (173h39m1.286989253s) [noise=true , address=77.90.31.14:4662 , piece_id=S5PPMCKIKSOCLUJ4CEH7ZDW3TVUZ7GTFFEIPCQK6FX25UIOUQHHQ, seg_pos={0 0}, upload=152907]: Long tail cancellation due to optimal threshold Nov 16 13:43:02.736224335 (173h39m1.287694796s) [noise=true , address=145.79.227.20:4437 , piece_id=G5P6OYXVAAQDZKTPPHDNZLMYTT6L2TFWKH7N4F7L6742BWHOQGBQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV): protocol: read tcp 188.166.225.170:35300->145.79.227.20:4437: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 13:43:02.736239203 (173h39m1.287709654s) [noise=true , address=145.79.227.20:4437 , piece_id=G5P6OYXVAAQDZKTPPHDNZLMYTT6L2TFWKH7N4F7L6742BWHOQGBQ, seg_pos={0 0}, upload=152907]: Long tail cancellation due to optimal threshold Nov 16 13:43:02.736270833 (173h39m1.287741294s) [seg_pos={0 0}, upload=152907]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 16 13:43:02.736437060 (173h39m1.287907521s) [upload=152907]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 16 13:43:02.736444273 (173h39m1.287914735s) [upload=152907]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 16 13:43:05.031396865 (173h39m3.582867336s) [upload=152907]: Done uploading segments... Nov 16 13:43:05.031422193 (173h39m3.582892654s) [upload=152907]: Upload finished. err: <nil> Duration: 6.065293925s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 10 out of 60 is failed (threshold=6) [.XX....X......X..XX....X......................X........X..X.] duration: 4.776844878s trace: https://jaeger.dev.storj.io/trace/3dcf6f8d25fcd7d8 FAIL (2025-11-16 14:14:35 UTC) orbiter warning: context deadline exceeded Duration: 4.776871438s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 6.000682429s trace: https://jaeger.dev.storj.io/trace/3c77028f820d12f9 FAIL (2025-11-16 14:13:51 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, Piece ID: UKP3MHXBSALCWFIHCTSR5JFFXEWWAWGDS4JYUYS2JEREQQKTPJSA) context deadline exceeded; piecestore: (Node ID: 12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, Piece ID: DNT5QVAMRO5OMBNVTDRRUKACMM7M5472LD5E4ASKYIBKBNWYAVRA) context deadline exceeded; piecestore: (Node ID: 17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, Piece ID: DPLALMJC4JPC5D7BOK5SOSDFDJFDDPGSJWP4XDDQE22AMFNAKATQ) context deadline exceeded ======LOGS====== gment. Inline: false Nov 16 13:42:52.666975419 (173h38m51.21844588s) [upload=152907]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 16 13:42:52.666980539 (173h38m51.21845099s) [upload=152907]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 16 13:42:53.024947049 (173h38m51.5764175s) [seg_pos={0 0}, upload=152907]: Begin upload segment... Nov 16 13:42:53.025007074 (173h38m51.576477535s) [seg_pos={0 0}, upload=152907]: Done begin upload segment. Nov 16 13:42:53.025087707 (173h38m51.576558158s) [noise=true , address=77.90.31.14:4662 , piece_id=S5PPMCKIKSOCLUJ4CEH7ZDW3TVUZ7GTFFEIPCQK6FX25UIOUQHHQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.025098377 (173h38m51.576568828s) [noise=true , address=145.79.227.31:4797 , piece_id=UKP3MHXBSALCWFIHCTSR5JFFXEWWAWGDS4JYUYS2JEREQQKTPJSA, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.025961451 (173h38m51.577431912s) [noise=true , address=45.91.116.6:4974 , piece_id=ZTS3H7E6A3VJECLRRKRKSVX5PH3VSH7F2HAUPQSPH56NCI6RTFBQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.026101888 (173h38m51.577572339s) [noise=true , address=145.79.227.10:4402 , piece_id=DNT5QVAMRO5OMBNVTDRRUKACMM7M5472LD5E4ASKYIBKBNWYAVRA, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.026397721 (173h38m51.577868182s) [noise=true , address=145.79.227.11:4732 , piece_id=GR6SASHNUTHAEOUPK7DWKVPHRGMG72FQMF4RINJ5NTIQQGFCVZZQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.026543650 (173h38m51.578014111s) [noise=true , address=145.79.227.20:4437 , piece_id=G5P6OYXVAAQDZKTPPHDNZLMYTT6L2TFWKH7N4F7L6742BWHOQGBQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.026887915 (173h38m51.578358366s) [noise=true , address=45.91.116.14:4189 , piece_id=NUMS6Q3F2OUAQ2PSKE2CZK2WKRQLZUZPWVBA3UA6UNKDQACPYSJQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.027039624 (173h38m51.578510085s) [noise=true , address=145.79.227.27:4691 , piece_id=NSX5S2D7ZSA5I57B4EAQEQXEVUMPKNMRKRV3RHWGWJ2DN7SWUPGQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.027374482 (173h38m51.578844973s) [noise=true , address=145.79.227.31:4798 , piece_id=DPLALMJC4JPC5D7BOK5SOSDFDJFDDPGSJWP4XDDQE22AMFNAKATQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.027472337 (173h38m51.578942798s) [noise=true , address=45.91.116.7:4010 , piece_id=EO7IEOCWEJAMZZU4BLIB5ALFACWTWZE3QCXU3FUY4FKC4P444ORA, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.027920821 (173h38m51.579391283s) [noise=true , address=145.79.227.23:4547 , piece_id=P4WX2ZDHV2Z6BFMF6IACIYW4DXKYFVSULUTF7EK2EMRO5OA5X5RQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.028194272 (173h38m51.579664743s) [noise=true , address=145.79.227.32:4834 , piece_id=CBK2UQ252EPFQDTOYS2X5WAUTJN4L2DTHGIOJXFZXWBOLGPRETXQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.028430262 (173h38m51.579900723s) [noise=true , address=145.79.227.9:4366 , piece_id=YFWF2YL2YPYVMP5DHJKRUGVUUS2ALG4DRJ3HUZCDZM4N3N676FNQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.029687726 (173h38m51.581158177s) [noise=true , address=145.79.227.20:4438 , piece_id=6HY2HDNJAZ552ZKXFJ7VNTQNBAQU5ZQLNKAG2NZZWZQHN6EEDG6Q, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.031784128 (173h38m51.583254589s) [noise=true , address=45.91.116.5:4298 , piece_id=ARGPAZE6CTMM6UXSMPSVHDBEQWKVGKY7WXOVLPB7UH3JZYMDB7JQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.034100830 (173h38m51.585571291s) [noise=true , address=77.90.31.10:4518 , piece_id=MSKQ66KE4WANAE4AWOD2IKY3PQN2YGZ5N4W3NXGKXSOLE7B6H6WQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:54.850200282 (173h38m53.401670733s) [upload=152907]: Next returned nil segment Nov 16 13:42:54.850219118 (173h38m53.401689569s) [upload=152907]: Waiting for error group managing segments... Nov 16 13:42:55.154333032 (173h38m53.705803493s) [noise=true , address=45.91.116.7:4010 , piece_id=EO7IEOCWEJAMZZU4BLIB5ALFACWTWZE3QCXU3FUY4FKC4P444ORA, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:42:55.201689040 (173h38m53.753159501s) [noise=true , address=77.90.31.10:4518 , piece_id=MSKQ66KE4WANAE4AWOD2IKY3PQN2YGZ5N4W3NXGKXSOLE7B6H6WQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:42:55.232313345 (173h38m53.783783816s) [noise=true , address=45.91.116.5:4298 , piece_id=ARGPAZE6CTMM6UXSMPSVHDBEQWKVGKY7WXOVLPB7UH3JZYMDB7JQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:42:55.260281393 (173h38m53.811751854s) [noise=true , address=45.91.116.6:4974 , piece_id=ZTS3H7E6A3VJECLRRKRKSVX5PH3VSH7F2HAUPQSPH56NCI6RTFBQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:00.229127085 (173h38m58.780597546s) [noise=true , address=145.79.227.20:4438 , piece_id=6HY2HDNJAZ552ZKXFJ7VNTQNBAQU5ZQLNKAG2NZZWZQHN6EEDG6Q, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:00.358241119 (173h38m58.909711581s) [noise=true , address=145.79.227.32:4834 , piece_id=CBK2UQ252EPFQDTOYS2X5WAUTJN4L2DTHGIOJXFZXWBOLGPRETXQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:00.646958215 (173h38m59.198428676s) [noise=true , address=145.79.227.31:4798 , piece_id=DPLALMJC4JPC5D7BOK5SOSDFDJFDDPGSJWP4XDDQE22AMFNAKATQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:00.908824080 (173h38m59.460294541s) [noise=true , address=145.79.227.23:4547 , piece_id=P4WX2ZDHV2Z6BFMF6IACIYW4DXKYFVSULUTF7EK2EMRO5OA5X5RQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:01.155346527 (173h38m59.706816989s) [noise=true , address=45.91.116.14:4189 , piece_id=NUMS6Q3F2OUAQ2PSKE2CZK2WKRQLZUZPWVBA3UA6UNKDQACPYSJQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:01.839444794 (173h39m0.390915255s) [noise=true , address=145.79.227.11:4732 , piece_id=GR6SASHNUTHAEOUPK7DWKVPHRGMG72FQMF4RINJ5NTIQQGFCVZZQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:02.591249579 (173h39m1.14272004s) [noise=true , address=145.79.227.10:4402 , piece_id=DNT5QVAMRO5OMBNVTDRRUKACMM7M5472LD5E4ASKYIBKBNWYAVRA, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:02.616126740 (173h39m1.167597202s) [noise=true , address=145.79.227.31:4797 , piece_id=UKP3MHXBSALCWFIHCTSR5JFFXEWWAWGDS4JYUYS2JEREQQKTPJSA, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:02.734605482 (173h39m1.286075943s) [noise=true , address=145.79.227.27:4691 , piece_id=NSX5S2D7ZSA5I57B4EAQEQXEVUMPKNMRKRV3RHWGWJ2DN7SWUPGQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:02.734638004 (173h39m1.286108475s) [seg_pos={0 0}, upload=152907]: Segment reached optimal threshold of 13 pieces. Nov 16 13:43:02.735049748 (173h39m1.286520209s) [noise=true , address=145.79.227.9:4366 , piece_id=YFWF2YL2YPYVMP5DHJKRUGVUUS2ALG4DRJ3HUZCDZM4N3N676FNQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy): protocol: write tcp 188.166.225.170:60100->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:60100->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60100->145.79.227.9:4366: use of closed network connection Nov 16 13:43:02.735060949 (173h39m1.28653141s) [noise=true , address=145.79.227.9:4366 , piece_id=YFWF2YL2YPYVMP5DHJKRUGVUUS2ALG4DRJ3HUZCDZM4N3N676FNQ, seg_pos={0 0}, upload=152907]: Long tail cancellation due to optimal threshold Nov 16 13:43:02.735509193 (173h39m1.286979654s) [noise=true , address=77.90.31.14:4662 , piece_id=S5PPMCKIKSOCLUJ4CEH7ZDW3TVUZ7GTFFEIPCQK6FX25UIOUQHHQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): protocol: write tcp 188.166.225.170:50196->77.90.31.14:4662: use of closed network connection; write tcp 188.166.225.170:50196->77.90.31.14:4662: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50196->77.90.31.14:4662: use of closed network connection Nov 16 13:43:02.735518792 (173h39m1.286989253s) [noise=true , address=77.90.31.14:4662 , piece_id=S5PPMCKIKSOCLUJ4CEH7ZDW3TVUZ7GTFFEIPCQK6FX25UIOUQHHQ, seg_pos={0 0}, upload=152907]: Long tail cancellation due to optimal threshold Nov 16 13:43:02.736224335 (173h39m1.287694796s) [noise=true , address=145.79.227.20:4437 , piece_id=G5P6OYXVAAQDZKTPPHDNZLMYTT6L2TFWKH7N4F7L6742BWHOQGBQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV): protocol: read tcp 188.166.225.170:35300->145.79.227.20:4437: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 13:43:02.736239203 (173h39m1.287709654s) [noise=true , address=145.79.227.20:4437 , piece_id=G5P6OYXVAAQDZKTPPHDNZLMYTT6L2TFWKH7N4F7L6742BWHOQGBQ, seg_pos={0 0}, upload=152907]: Long tail cancellation due to optimal threshold Nov 16 13:43:02.736270833 (173h39m1.287741294s) [seg_pos={0 0}, upload=152907]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 16 13:43:02.736437060 (173h39m1.287907521s) [upload=152907]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 16 13:43:02.736444273 (173h39m1.287914735s) [upload=152907]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 16 13:43:05.031396865 (173h39m3.582867336s) [upload=152907]: Done uploading segments... Nov 16 13:43:05.031422193 (173h39m3.582892654s) [upload=152907]: Upload finished. err: <nil> Duration: 6.000717005s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 10 out of 60 is failed (threshold=6) [..XX....X......X..XX....X......................X........X..X] duration: 5.000857237s trace: https://jaeger.dev.storj.io/trace/3dcf6f8d25fcd7d8 FAIL (2025-11-16 14:13:25 UTC) orbiter warning: context deadline exceeded Duration: 5.000893905s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 6.149076416s trace: https://jaeger.dev.storj.io/trace/3c77028f820d12f9 FAIL (2025-11-16 14:13:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, Piece ID: UKP3MHXBSALCWFIHCTSR5JFFXEWWAWGDS4JYUYS2JEREQQKTPJSA) context deadline exceeded; piecestore: (Node ID: 12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, Piece ID: DNT5QVAMRO5OMBNVTDRRUKACMM7M5472LD5E4ASKYIBKBNWYAVRA) context deadline exceeded; piecestore: (Node ID: 17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, Piece ID: DPLALMJC4JPC5D7BOK5SOSDFDJFDDPGSJWP4XDDQE22AMFNAKATQ) context deadline exceeded ======LOGS====== gment. Inline: false Nov 16 13:42:52.666975419 (173h38m51.21844588s) [upload=152907]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 16 13:42:52.666980539 (173h38m51.21845099s) [upload=152907]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 16 13:42:53.024947049 (173h38m51.5764175s) [seg_pos={0 0}, upload=152907]: Begin upload segment... Nov 16 13:42:53.025007074 (173h38m51.576477535s) [seg_pos={0 0}, upload=152907]: Done begin upload segment. Nov 16 13:42:53.025087707 (173h38m51.576558158s) [noise=true , address=77.90.31.14:4662 , piece_id=S5PPMCKIKSOCLUJ4CEH7ZDW3TVUZ7GTFFEIPCQK6FX25UIOUQHHQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.025098377 (173h38m51.576568828s) [noise=true , address=145.79.227.31:4797 , piece_id=UKP3MHXBSALCWFIHCTSR5JFFXEWWAWGDS4JYUYS2JEREQQKTPJSA, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.025961451 (173h38m51.577431912s) [noise=true , address=45.91.116.6:4974 , piece_id=ZTS3H7E6A3VJECLRRKRKSVX5PH3VSH7F2HAUPQSPH56NCI6RTFBQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.026101888 (173h38m51.577572339s) [noise=true , address=145.79.227.10:4402 , piece_id=DNT5QVAMRO5OMBNVTDRRUKACMM7M5472LD5E4ASKYIBKBNWYAVRA, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.026397721 (173h38m51.577868182s) [noise=true , address=145.79.227.11:4732 , piece_id=GR6SASHNUTHAEOUPK7DWKVPHRGMG72FQMF4RINJ5NTIQQGFCVZZQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.026543650 (173h38m51.578014111s) [noise=true , address=145.79.227.20:4437 , piece_id=G5P6OYXVAAQDZKTPPHDNZLMYTT6L2TFWKH7N4F7L6742BWHOQGBQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.026887915 (173h38m51.578358366s) [noise=true , address=45.91.116.14:4189 , piece_id=NUMS6Q3F2OUAQ2PSKE2CZK2WKRQLZUZPWVBA3UA6UNKDQACPYSJQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.027039624 (173h38m51.578510085s) [noise=true , address=145.79.227.27:4691 , piece_id=NSX5S2D7ZSA5I57B4EAQEQXEVUMPKNMRKRV3RHWGWJ2DN7SWUPGQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.027374482 (173h38m51.578844973s) [noise=true , address=145.79.227.31:4798 , piece_id=DPLALMJC4JPC5D7BOK5SOSDFDJFDDPGSJWP4XDDQE22AMFNAKATQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.027472337 (173h38m51.578942798s) [noise=true , address=45.91.116.7:4010 , piece_id=EO7IEOCWEJAMZZU4BLIB5ALFACWTWZE3QCXU3FUY4FKC4P444ORA, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.027920821 (173h38m51.579391283s) [noise=true , address=145.79.227.23:4547 , piece_id=P4WX2ZDHV2Z6BFMF6IACIYW4DXKYFVSULUTF7EK2EMRO5OA5X5RQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.028194272 (173h38m51.579664743s) [noise=true , address=145.79.227.32:4834 , piece_id=CBK2UQ252EPFQDTOYS2X5WAUTJN4L2DTHGIOJXFZXWBOLGPRETXQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.028430262 (173h38m51.579900723s) [noise=true , address=145.79.227.9:4366 , piece_id=YFWF2YL2YPYVMP5DHJKRUGVUUS2ALG4DRJ3HUZCDZM4N3N676FNQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.029687726 (173h38m51.581158177s) [noise=true , address=145.79.227.20:4438 , piece_id=6HY2HDNJAZ552ZKXFJ7VNTQNBAQU5ZQLNKAG2NZZWZQHN6EEDG6Q, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.031784128 (173h38m51.583254589s) [noise=true , address=45.91.116.5:4298 , piece_id=ARGPAZE6CTMM6UXSMPSVHDBEQWKVGKY7WXOVLPB7UH3JZYMDB7JQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.034100830 (173h38m51.585571291s) [noise=true , address=77.90.31.10:4518 , piece_id=MSKQ66KE4WANAE4AWOD2IKY3PQN2YGZ5N4W3NXGKXSOLE7B6H6WQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:54.850200282 (173h38m53.401670733s) [upload=152907]: Next returned nil segment Nov 16 13:42:54.850219118 (173h38m53.401689569s) [upload=152907]: Waiting for error group managing segments... Nov 16 13:42:55.154333032 (173h38m53.705803493s) [noise=true , address=45.91.116.7:4010 , piece_id=EO7IEOCWEJAMZZU4BLIB5ALFACWTWZE3QCXU3FUY4FKC4P444ORA, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:42:55.201689040 (173h38m53.753159501s) [noise=true , address=77.90.31.10:4518 , piece_id=MSKQ66KE4WANAE4AWOD2IKY3PQN2YGZ5N4W3NXGKXSOLE7B6H6WQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:42:55.232313345 (173h38m53.783783816s) [noise=true , address=45.91.116.5:4298 , piece_id=ARGPAZE6CTMM6UXSMPSVHDBEQWKVGKY7WXOVLPB7UH3JZYMDB7JQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:42:55.260281393 (173h38m53.811751854s) [noise=true , address=45.91.116.6:4974 , piece_id=ZTS3H7E6A3VJECLRRKRKSVX5PH3VSH7F2HAUPQSPH56NCI6RTFBQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:00.229127085 (173h38m58.780597546s) [noise=true , address=145.79.227.20:4438 , piece_id=6HY2HDNJAZ552ZKXFJ7VNTQNBAQU5ZQLNKAG2NZZWZQHN6EEDG6Q, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:00.358241119 (173h38m58.909711581s) [noise=true , address=145.79.227.32:4834 , piece_id=CBK2UQ252EPFQDTOYS2X5WAUTJN4L2DTHGIOJXFZXWBOLGPRETXQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:00.646958215 (173h38m59.198428676s) [noise=true , address=145.79.227.31:4798 , piece_id=DPLALMJC4JPC5D7BOK5SOSDFDJFDDPGSJWP4XDDQE22AMFNAKATQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:00.908824080 (173h38m59.460294541s) [noise=true , address=145.79.227.23:4547 , piece_id=P4WX2ZDHV2Z6BFMF6IACIYW4DXKYFVSULUTF7EK2EMRO5OA5X5RQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:01.155346527 (173h38m59.706816989s) [noise=true , address=45.91.116.14:4189 , piece_id=NUMS6Q3F2OUAQ2PSKE2CZK2WKRQLZUZPWVBA3UA6UNKDQACPYSJQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:01.839444794 (173h39m0.390915255s) [noise=true , address=145.79.227.11:4732 , piece_id=GR6SASHNUTHAEOUPK7DWKVPHRGMG72FQMF4RINJ5NTIQQGFCVZZQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:02.591249579 (173h39m1.14272004s) [noise=true , address=145.79.227.10:4402 , piece_id=DNT5QVAMRO5OMBNVTDRRUKACMM7M5472LD5E4ASKYIBKBNWYAVRA, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:02.616126740 (173h39m1.167597202s) [noise=true , address=145.79.227.31:4797 , piece_id=UKP3MHXBSALCWFIHCTSR5JFFXEWWAWGDS4JYUYS2JEREQQKTPJSA, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:02.734605482 (173h39m1.286075943s) [noise=true , address=145.79.227.27:4691 , piece_id=NSX5S2D7ZSA5I57B4EAQEQXEVUMPKNMRKRV3RHWGWJ2DN7SWUPGQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:02.734638004 (173h39m1.286108475s) [seg_pos={0 0}, upload=152907]: Segment reached optimal threshold of 13 pieces. Nov 16 13:43:02.735049748 (173h39m1.286520209s) [noise=true , address=145.79.227.9:4366 , piece_id=YFWF2YL2YPYVMP5DHJKRUGVUUS2ALG4DRJ3HUZCDZM4N3N676FNQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy): protocol: write tcp 188.166.225.170:60100->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:60100->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60100->145.79.227.9:4366: use of closed network connection Nov 16 13:43:02.735060949 (173h39m1.28653141s) [noise=true , address=145.79.227.9:4366 , piece_id=YFWF2YL2YPYVMP5DHJKRUGVUUS2ALG4DRJ3HUZCDZM4N3N676FNQ, seg_pos={0 0}, upload=152907]: Long tail cancellation due to optimal threshold Nov 16 13:43:02.735509193 (173h39m1.286979654s) [noise=true , address=77.90.31.14:4662 , piece_id=S5PPMCKIKSOCLUJ4CEH7ZDW3TVUZ7GTFFEIPCQK6FX25UIOUQHHQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): protocol: write tcp 188.166.225.170:50196->77.90.31.14:4662: use of closed network connection; write tcp 188.166.225.170:50196->77.90.31.14:4662: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50196->77.90.31.14:4662: use of closed network connection Nov 16 13:43:02.735518792 (173h39m1.286989253s) [noise=true , address=77.90.31.14:4662 , piece_id=S5PPMCKIKSOCLUJ4CEH7ZDW3TVUZ7GTFFEIPCQK6FX25UIOUQHHQ, seg_pos={0 0}, upload=152907]: Long tail cancellation due to optimal threshold Nov 16 13:43:02.736224335 (173h39m1.287694796s) [noise=true , address=145.79.227.20:4437 , piece_id=G5P6OYXVAAQDZKTPPHDNZLMYTT6L2TFWKH7N4F7L6742BWHOQGBQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV): protocol: read tcp 188.166.225.170:35300->145.79.227.20:4437: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 13:43:02.736239203 (173h39m1.287709654s) [noise=true , address=145.79.227.20:4437 , piece_id=G5P6OYXVAAQDZKTPPHDNZLMYTT6L2TFWKH7N4F7L6742BWHOQGBQ, seg_pos={0 0}, upload=152907]: Long tail cancellation due to optimal threshold Nov 16 13:43:02.736270833 (173h39m1.287741294s) [seg_pos={0 0}, upload=152907]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 16 13:43:02.736437060 (173h39m1.287907521s) [upload=152907]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 16 13:43:02.736444273 (173h39m1.287914735s) [upload=152907]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 16 13:43:05.031396865 (173h39m3.582867336s) [upload=152907]: Done uploading segments... Nov 16 13:43:05.031422193 (173h39m3.582892654s) [upload=152907]: Upload finished. err: <nil> Duration: 6.149111342s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 9 out of 60 is failed (threshold=6) [...XX....X......X..XX....X......................X........X..] duration: 5.000481442s trace: https://jaeger.dev.storj.io/trace/771941b4ae81b918 FAIL (2025-11-16 14:12:36 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000574909s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 5.978950037s trace: https://jaeger.dev.storj.io/trace/3c77028f820d12f9 FAIL (2025-11-16 14:12:13 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, Piece ID: UKP3MHXBSALCWFIHCTSR5JFFXEWWAWGDS4JYUYS2JEREQQKTPJSA) context deadline exceeded; piecestore: (Node ID: 12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, Piece ID: DNT5QVAMRO5OMBNVTDRRUKACMM7M5472LD5E4ASKYIBKBNWYAVRA) context deadline exceeded; piecestore: (Node ID: 17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, Piece ID: DPLALMJC4JPC5D7BOK5SOSDFDJFDDPGSJWP4XDDQE22AMFNAKATQ) context deadline exceeded ======LOGS====== gment. Inline: false Nov 16 13:42:52.666975419 (173h38m51.21844588s) [upload=152907]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 16 13:42:52.666980539 (173h38m51.21845099s) [upload=152907]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 16 13:42:53.024947049 (173h38m51.5764175s) [seg_pos={0 0}, upload=152907]: Begin upload segment... Nov 16 13:42:53.025007074 (173h38m51.576477535s) [seg_pos={0 0}, upload=152907]: Done begin upload segment. Nov 16 13:42:53.025087707 (173h38m51.576558158s) [noise=true , address=77.90.31.14:4662 , piece_id=S5PPMCKIKSOCLUJ4CEH7ZDW3TVUZ7GTFFEIPCQK6FX25UIOUQHHQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.025098377 (173h38m51.576568828s) [noise=true , address=145.79.227.31:4797 , piece_id=UKP3MHXBSALCWFIHCTSR5JFFXEWWAWGDS4JYUYS2JEREQQKTPJSA, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.025961451 (173h38m51.577431912s) [noise=true , address=45.91.116.6:4974 , piece_id=ZTS3H7E6A3VJECLRRKRKSVX5PH3VSH7F2HAUPQSPH56NCI6RTFBQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.026101888 (173h38m51.577572339s) [noise=true , address=145.79.227.10:4402 , piece_id=DNT5QVAMRO5OMBNVTDRRUKACMM7M5472LD5E4ASKYIBKBNWYAVRA, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.026397721 (173h38m51.577868182s) [noise=true , address=145.79.227.11:4732 , piece_id=GR6SASHNUTHAEOUPK7DWKVPHRGMG72FQMF4RINJ5NTIQQGFCVZZQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.026543650 (173h38m51.578014111s) [noise=true , address=145.79.227.20:4437 , piece_id=G5P6OYXVAAQDZKTPPHDNZLMYTT6L2TFWKH7N4F7L6742BWHOQGBQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.026887915 (173h38m51.578358366s) [noise=true , address=45.91.116.14:4189 , piece_id=NUMS6Q3F2OUAQ2PSKE2CZK2WKRQLZUZPWVBA3UA6UNKDQACPYSJQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.027039624 (173h38m51.578510085s) [noise=true , address=145.79.227.27:4691 , piece_id=NSX5S2D7ZSA5I57B4EAQEQXEVUMPKNMRKRV3RHWGWJ2DN7SWUPGQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.027374482 (173h38m51.578844973s) [noise=true , address=145.79.227.31:4798 , piece_id=DPLALMJC4JPC5D7BOK5SOSDFDJFDDPGSJWP4XDDQE22AMFNAKATQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.027472337 (173h38m51.578942798s) [noise=true , address=45.91.116.7:4010 , piece_id=EO7IEOCWEJAMZZU4BLIB5ALFACWTWZE3QCXU3FUY4FKC4P444ORA, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.027920821 (173h38m51.579391283s) [noise=true , address=145.79.227.23:4547 , piece_id=P4WX2ZDHV2Z6BFMF6IACIYW4DXKYFVSULUTF7EK2EMRO5OA5X5RQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.028194272 (173h38m51.579664743s) [noise=true , address=145.79.227.32:4834 , piece_id=CBK2UQ252EPFQDTOYS2X5WAUTJN4L2DTHGIOJXFZXWBOLGPRETXQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.028430262 (173h38m51.579900723s) [noise=true , address=145.79.227.9:4366 , piece_id=YFWF2YL2YPYVMP5DHJKRUGVUUS2ALG4DRJ3HUZCDZM4N3N676FNQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.029687726 (173h38m51.581158177s) [noise=true , address=145.79.227.20:4438 , piece_id=6HY2HDNJAZ552ZKXFJ7VNTQNBAQU5ZQLNKAG2NZZWZQHN6EEDG6Q, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.031784128 (173h38m51.583254589s) [noise=true , address=45.91.116.5:4298 , piece_id=ARGPAZE6CTMM6UXSMPSVHDBEQWKVGKY7WXOVLPB7UH3JZYMDB7JQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.034100830 (173h38m51.585571291s) [noise=true , address=77.90.31.10:4518 , piece_id=MSKQ66KE4WANAE4AWOD2IKY3PQN2YGZ5N4W3NXGKXSOLE7B6H6WQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:54.850200282 (173h38m53.401670733s) [upload=152907]: Next returned nil segment Nov 16 13:42:54.850219118 (173h38m53.401689569s) [upload=152907]: Waiting for error group managing segments... Nov 16 13:42:55.154333032 (173h38m53.705803493s) [noise=true , address=45.91.116.7:4010 , piece_id=EO7IEOCWEJAMZZU4BLIB5ALFACWTWZE3QCXU3FUY4FKC4P444ORA, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:42:55.201689040 (173h38m53.753159501s) [noise=true , address=77.90.31.10:4518 , piece_id=MSKQ66KE4WANAE4AWOD2IKY3PQN2YGZ5N4W3NXGKXSOLE7B6H6WQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:42:55.232313345 (173h38m53.783783816s) [noise=true , address=45.91.116.5:4298 , piece_id=ARGPAZE6CTMM6UXSMPSVHDBEQWKVGKY7WXOVLPB7UH3JZYMDB7JQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:42:55.260281393 (173h38m53.811751854s) [noise=true , address=45.91.116.6:4974 , piece_id=ZTS3H7E6A3VJECLRRKRKSVX5PH3VSH7F2HAUPQSPH56NCI6RTFBQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:00.229127085 (173h38m58.780597546s) [noise=true , address=145.79.227.20:4438 , piece_id=6HY2HDNJAZ552ZKXFJ7VNTQNBAQU5ZQLNKAG2NZZWZQHN6EEDG6Q, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:00.358241119 (173h38m58.909711581s) [noise=true , address=145.79.227.32:4834 , piece_id=CBK2UQ252EPFQDTOYS2X5WAUTJN4L2DTHGIOJXFZXWBOLGPRETXQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:00.646958215 (173h38m59.198428676s) [noise=true , address=145.79.227.31:4798 , piece_id=DPLALMJC4JPC5D7BOK5SOSDFDJFDDPGSJWP4XDDQE22AMFNAKATQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:00.908824080 (173h38m59.460294541s) [noise=true , address=145.79.227.23:4547 , piece_id=P4WX2ZDHV2Z6BFMF6IACIYW4DXKYFVSULUTF7EK2EMRO5OA5X5RQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:01.155346527 (173h38m59.706816989s) [noise=true , address=45.91.116.14:4189 , piece_id=NUMS6Q3F2OUAQ2PSKE2CZK2WKRQLZUZPWVBA3UA6UNKDQACPYSJQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:01.839444794 (173h39m0.390915255s) [noise=true , address=145.79.227.11:4732 , piece_id=GR6SASHNUTHAEOUPK7DWKVPHRGMG72FQMF4RINJ5NTIQQGFCVZZQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:02.591249579 (173h39m1.14272004s) [noise=true , address=145.79.227.10:4402 , piece_id=DNT5QVAMRO5OMBNVTDRRUKACMM7M5472LD5E4ASKYIBKBNWYAVRA, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:02.616126740 (173h39m1.167597202s) [noise=true , address=145.79.227.31:4797 , piece_id=UKP3MHXBSALCWFIHCTSR5JFFXEWWAWGDS4JYUYS2JEREQQKTPJSA, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:02.734605482 (173h39m1.286075943s) [noise=true , address=145.79.227.27:4691 , piece_id=NSX5S2D7ZSA5I57B4EAQEQXEVUMPKNMRKRV3RHWGWJ2DN7SWUPGQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:02.734638004 (173h39m1.286108475s) [seg_pos={0 0}, upload=152907]: Segment reached optimal threshold of 13 pieces. Nov 16 13:43:02.735049748 (173h39m1.286520209s) [noise=true , address=145.79.227.9:4366 , piece_id=YFWF2YL2YPYVMP5DHJKRUGVUUS2ALG4DRJ3HUZCDZM4N3N676FNQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy): protocol: write tcp 188.166.225.170:60100->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:60100->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60100->145.79.227.9:4366: use of closed network connection Nov 16 13:43:02.735060949 (173h39m1.28653141s) [noise=true , address=145.79.227.9:4366 , piece_id=YFWF2YL2YPYVMP5DHJKRUGVUUS2ALG4DRJ3HUZCDZM4N3N676FNQ, seg_pos={0 0}, upload=152907]: Long tail cancellation due to optimal threshold Nov 16 13:43:02.735509193 (173h39m1.286979654s) [noise=true , address=77.90.31.14:4662 , piece_id=S5PPMCKIKSOCLUJ4CEH7ZDW3TVUZ7GTFFEIPCQK6FX25UIOUQHHQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): protocol: write tcp 188.166.225.170:50196->77.90.31.14:4662: use of closed network connection; write tcp 188.166.225.170:50196->77.90.31.14:4662: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50196->77.90.31.14:4662: use of closed network connection Nov 16 13:43:02.735518792 (173h39m1.286989253s) [noise=true , address=77.90.31.14:4662 , piece_id=S5PPMCKIKSOCLUJ4CEH7ZDW3TVUZ7GTFFEIPCQK6FX25UIOUQHHQ, seg_pos={0 0}, upload=152907]: Long tail cancellation due to optimal threshold Nov 16 13:43:02.736224335 (173h39m1.287694796s) [noise=true , address=145.79.227.20:4437 , piece_id=G5P6OYXVAAQDZKTPPHDNZLMYTT6L2TFWKH7N4F7L6742BWHOQGBQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV): protocol: read tcp 188.166.225.170:35300->145.79.227.20:4437: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 13:43:02.736239203 (173h39m1.287709654s) [noise=true , address=145.79.227.20:4437 , piece_id=G5P6OYXVAAQDZKTPPHDNZLMYTT6L2TFWKH7N4F7L6742BWHOQGBQ, seg_pos={0 0}, upload=152907]: Long tail cancellation due to optimal threshold Nov 16 13:43:02.736270833 (173h39m1.287741294s) [seg_pos={0 0}, upload=152907]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 16 13:43:02.736437060 (173h39m1.287907521s) [upload=152907]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 16 13:43:02.736444273 (173h39m1.287914735s) [upload=152907]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 16 13:43:05.031396865 (173h39m3.582867336s) [upload=152907]: Done uploading segments... Nov 16 13:43:05.031422193 (173h39m3.582892654s) [upload=152907]: Upload finished. err: <nil> Duration: 5.97896702s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 9 out of 60 is failed (threshold=6) [....XX....X......X..XX....X......................X........X.] duration: 4.24880391s trace: https://jaeger.dev.storj.io/trace/771941b4ae81b918 FAIL (2025-11-16 14:11:22 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.248831772s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 5.988336941s trace: https://jaeger.dev.storj.io/trace/3c77028f820d12f9 FAIL (2025-11-16 14:11:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, Piece ID: UKP3MHXBSALCWFIHCTSR5JFFXEWWAWGDS4JYUYS2JEREQQKTPJSA) context deadline exceeded; piecestore: (Node ID: 12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, Piece ID: DNT5QVAMRO5OMBNVTDRRUKACMM7M5472LD5E4ASKYIBKBNWYAVRA) context deadline exceeded; piecestore: (Node ID: 17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, Piece ID: DPLALMJC4JPC5D7BOK5SOSDFDJFDDPGSJWP4XDDQE22AMFNAKATQ) context deadline exceeded ======LOGS====== gment. Inline: false Nov 16 13:42:52.666975419 (173h38m51.21844588s) [upload=152907]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 16 13:42:52.666980539 (173h38m51.21845099s) [upload=152907]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 16 13:42:53.024947049 (173h38m51.5764175s) [seg_pos={0 0}, upload=152907]: Begin upload segment... Nov 16 13:42:53.025007074 (173h38m51.576477535s) [seg_pos={0 0}, upload=152907]: Done begin upload segment. Nov 16 13:42:53.025087707 (173h38m51.576558158s) [noise=true , address=77.90.31.14:4662 , piece_id=S5PPMCKIKSOCLUJ4CEH7ZDW3TVUZ7GTFFEIPCQK6FX25UIOUQHHQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.025098377 (173h38m51.576568828s) [noise=true , address=145.79.227.31:4797 , piece_id=UKP3MHXBSALCWFIHCTSR5JFFXEWWAWGDS4JYUYS2JEREQQKTPJSA, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.025961451 (173h38m51.577431912s) [noise=true , address=45.91.116.6:4974 , piece_id=ZTS3H7E6A3VJECLRRKRKSVX5PH3VSH7F2HAUPQSPH56NCI6RTFBQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.026101888 (173h38m51.577572339s) [noise=true , address=145.79.227.10:4402 , piece_id=DNT5QVAMRO5OMBNVTDRRUKACMM7M5472LD5E4ASKYIBKBNWYAVRA, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.026397721 (173h38m51.577868182s) [noise=true , address=145.79.227.11:4732 , piece_id=GR6SASHNUTHAEOUPK7DWKVPHRGMG72FQMF4RINJ5NTIQQGFCVZZQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.026543650 (173h38m51.578014111s) [noise=true , address=145.79.227.20:4437 , piece_id=G5P6OYXVAAQDZKTPPHDNZLMYTT6L2TFWKH7N4F7L6742BWHOQGBQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.026887915 (173h38m51.578358366s) [noise=true , address=45.91.116.14:4189 , piece_id=NUMS6Q3F2OUAQ2PSKE2CZK2WKRQLZUZPWVBA3UA6UNKDQACPYSJQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.027039624 (173h38m51.578510085s) [noise=true , address=145.79.227.27:4691 , piece_id=NSX5S2D7ZSA5I57B4EAQEQXEVUMPKNMRKRV3RHWGWJ2DN7SWUPGQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.027374482 (173h38m51.578844973s) [noise=true , address=145.79.227.31:4798 , piece_id=DPLALMJC4JPC5D7BOK5SOSDFDJFDDPGSJWP4XDDQE22AMFNAKATQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.027472337 (173h38m51.578942798s) [noise=true , address=45.91.116.7:4010 , piece_id=EO7IEOCWEJAMZZU4BLIB5ALFACWTWZE3QCXU3FUY4FKC4P444ORA, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.027920821 (173h38m51.579391283s) [noise=true , address=145.79.227.23:4547 , piece_id=P4WX2ZDHV2Z6BFMF6IACIYW4DXKYFVSULUTF7EK2EMRO5OA5X5RQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.028194272 (173h38m51.579664743s) [noise=true , address=145.79.227.32:4834 , piece_id=CBK2UQ252EPFQDTOYS2X5WAUTJN4L2DTHGIOJXFZXWBOLGPRETXQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.028430262 (173h38m51.579900723s) [noise=true , address=145.79.227.9:4366 , piece_id=YFWF2YL2YPYVMP5DHJKRUGVUUS2ALG4DRJ3HUZCDZM4N3N676FNQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.029687726 (173h38m51.581158177s) [noise=true , address=145.79.227.20:4438 , piece_id=6HY2HDNJAZ552ZKXFJ7VNTQNBAQU5ZQLNKAG2NZZWZQHN6EEDG6Q, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.031784128 (173h38m51.583254589s) [noise=true , address=45.91.116.5:4298 , piece_id=ARGPAZE6CTMM6UXSMPSVHDBEQWKVGKY7WXOVLPB7UH3JZYMDB7JQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.034100830 (173h38m51.585571291s) [noise=true , address=77.90.31.10:4518 , piece_id=MSKQ66KE4WANAE4AWOD2IKY3PQN2YGZ5N4W3NXGKXSOLE7B6H6WQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:54.850200282 (173h38m53.401670733s) [upload=152907]: Next returned nil segment Nov 16 13:42:54.850219118 (173h38m53.401689569s) [upload=152907]: Waiting for error group managing segments... Nov 16 13:42:55.154333032 (173h38m53.705803493s) [noise=true , address=45.91.116.7:4010 , piece_id=EO7IEOCWEJAMZZU4BLIB5ALFACWTWZE3QCXU3FUY4FKC4P444ORA, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:42:55.201689040 (173h38m53.753159501s) [noise=true , address=77.90.31.10:4518 , piece_id=MSKQ66KE4WANAE4AWOD2IKY3PQN2YGZ5N4W3NXGKXSOLE7B6H6WQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:42:55.232313345 (173h38m53.783783816s) [noise=true , address=45.91.116.5:4298 , piece_id=ARGPAZE6CTMM6UXSMPSVHDBEQWKVGKY7WXOVLPB7UH3JZYMDB7JQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:42:55.260281393 (173h38m53.811751854s) [noise=true , address=45.91.116.6:4974 , piece_id=ZTS3H7E6A3VJECLRRKRKSVX5PH3VSH7F2HAUPQSPH56NCI6RTFBQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:00.229127085 (173h38m58.780597546s) [noise=true , address=145.79.227.20:4438 , piece_id=6HY2HDNJAZ552ZKXFJ7VNTQNBAQU5ZQLNKAG2NZZWZQHN6EEDG6Q, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:00.358241119 (173h38m58.909711581s) [noise=true , address=145.79.227.32:4834 , piece_id=CBK2UQ252EPFQDTOYS2X5WAUTJN4L2DTHGIOJXFZXWBOLGPRETXQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:00.646958215 (173h38m59.198428676s) [noise=true , address=145.79.227.31:4798 , piece_id=DPLALMJC4JPC5D7BOK5SOSDFDJFDDPGSJWP4XDDQE22AMFNAKATQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:00.908824080 (173h38m59.460294541s) [noise=true , address=145.79.227.23:4547 , piece_id=P4WX2ZDHV2Z6BFMF6IACIYW4DXKYFVSULUTF7EK2EMRO5OA5X5RQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:01.155346527 (173h38m59.706816989s) [noise=true , address=45.91.116.14:4189 , piece_id=NUMS6Q3F2OUAQ2PSKE2CZK2WKRQLZUZPWVBA3UA6UNKDQACPYSJQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:01.839444794 (173h39m0.390915255s) [noise=true , address=145.79.227.11:4732 , piece_id=GR6SASHNUTHAEOUPK7DWKVPHRGMG72FQMF4RINJ5NTIQQGFCVZZQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:02.591249579 (173h39m1.14272004s) [noise=true , address=145.79.227.10:4402 , piece_id=DNT5QVAMRO5OMBNVTDRRUKACMM7M5472LD5E4ASKYIBKBNWYAVRA, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:02.616126740 (173h39m1.167597202s) [noise=true , address=145.79.227.31:4797 , piece_id=UKP3MHXBSALCWFIHCTSR5JFFXEWWAWGDS4JYUYS2JEREQQKTPJSA, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:02.734605482 (173h39m1.286075943s) [noise=true , address=145.79.227.27:4691 , piece_id=NSX5S2D7ZSA5I57B4EAQEQXEVUMPKNMRKRV3RHWGWJ2DN7SWUPGQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:02.734638004 (173h39m1.286108475s) [seg_pos={0 0}, upload=152907]: Segment reached optimal threshold of 13 pieces. Nov 16 13:43:02.735049748 (173h39m1.286520209s) [noise=true , address=145.79.227.9:4366 , piece_id=YFWF2YL2YPYVMP5DHJKRUGVUUS2ALG4DRJ3HUZCDZM4N3N676FNQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy): protocol: write tcp 188.166.225.170:60100->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:60100->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60100->145.79.227.9:4366: use of closed network connection Nov 16 13:43:02.735060949 (173h39m1.28653141s) [noise=true , address=145.79.227.9:4366 , piece_id=YFWF2YL2YPYVMP5DHJKRUGVUUS2ALG4DRJ3HUZCDZM4N3N676FNQ, seg_pos={0 0}, upload=152907]: Long tail cancellation due to optimal threshold Nov 16 13:43:02.735509193 (173h39m1.286979654s) [noise=true , address=77.90.31.14:4662 , piece_id=S5PPMCKIKSOCLUJ4CEH7ZDW3TVUZ7GTFFEIPCQK6FX25UIOUQHHQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): protocol: write tcp 188.166.225.170:50196->77.90.31.14:4662: use of closed network connection; write tcp 188.166.225.170:50196->77.90.31.14:4662: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50196->77.90.31.14:4662: use of closed network connection Nov 16 13:43:02.735518792 (173h39m1.286989253s) [noise=true , address=77.90.31.14:4662 , piece_id=S5PPMCKIKSOCLUJ4CEH7ZDW3TVUZ7GTFFEIPCQK6FX25UIOUQHHQ, seg_pos={0 0}, upload=152907]: Long tail cancellation due to optimal threshold Nov 16 13:43:02.736224335 (173h39m1.287694796s) [noise=true , address=145.79.227.20:4437 , piece_id=G5P6OYXVAAQDZKTPPHDNZLMYTT6L2TFWKH7N4F7L6742BWHOQGBQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV): protocol: read tcp 188.166.225.170:35300->145.79.227.20:4437: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 13:43:02.736239203 (173h39m1.287709654s) [noise=true , address=145.79.227.20:4437 , piece_id=G5P6OYXVAAQDZKTPPHDNZLMYTT6L2TFWKH7N4F7L6742BWHOQGBQ, seg_pos={0 0}, upload=152907]: Long tail cancellation due to optimal threshold Nov 16 13:43:02.736270833 (173h39m1.287741294s) [seg_pos={0 0}, upload=152907]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 16 13:43:02.736437060 (173h39m1.287907521s) [upload=152907]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 16 13:43:02.736444273 (173h39m1.287914735s) [upload=152907]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 16 13:43:05.031396865 (173h39m3.582867336s) [upload=152907]: Done uploading segments... Nov 16 13:43:05.031422193 (173h39m3.582892654s) [upload=152907]: Upload finished. err: <nil> Duration: 5.98835232s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 5.991891197s trace: https://jaeger.dev.storj.io/trace/3c77028f820d12f9 FAIL (2025-11-16 14:10:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, Piece ID: UKP3MHXBSALCWFIHCTSR5JFFXEWWAWGDS4JYUYS2JEREQQKTPJSA) context deadline exceeded; piecestore: (Node ID: 12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, Piece ID: DNT5QVAMRO5OMBNVTDRRUKACMM7M5472LD5E4ASKYIBKBNWYAVRA) context deadline exceeded; piecestore: (Node ID: 17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, Piece ID: DPLALMJC4JPC5D7BOK5SOSDFDJFDDPGSJWP4XDDQE22AMFNAKATQ) context deadline exceeded ======LOGS====== gment. Inline: false Nov 16 13:42:52.666975419 (173h38m51.21844588s) [upload=152907]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 16 13:42:52.666980539 (173h38m51.21845099s) [upload=152907]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 16 13:42:53.024947049 (173h38m51.5764175s) [seg_pos={0 0}, upload=152907]: Begin upload segment... Nov 16 13:42:53.025007074 (173h38m51.576477535s) [seg_pos={0 0}, upload=152907]: Done begin upload segment. Nov 16 13:42:53.025087707 (173h38m51.576558158s) [noise=true , address=77.90.31.14:4662 , piece_id=S5PPMCKIKSOCLUJ4CEH7ZDW3TVUZ7GTFFEIPCQK6FX25UIOUQHHQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.025098377 (173h38m51.576568828s) [noise=true , address=145.79.227.31:4797 , piece_id=UKP3MHXBSALCWFIHCTSR5JFFXEWWAWGDS4JYUYS2JEREQQKTPJSA, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.025961451 (173h38m51.577431912s) [noise=true , address=45.91.116.6:4974 , piece_id=ZTS3H7E6A3VJECLRRKRKSVX5PH3VSH7F2HAUPQSPH56NCI6RTFBQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.026101888 (173h38m51.577572339s) [noise=true , address=145.79.227.10:4402 , piece_id=DNT5QVAMRO5OMBNVTDRRUKACMM7M5472LD5E4ASKYIBKBNWYAVRA, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.026397721 (173h38m51.577868182s) [noise=true , address=145.79.227.11:4732 , piece_id=GR6SASHNUTHAEOUPK7DWKVPHRGMG72FQMF4RINJ5NTIQQGFCVZZQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.026543650 (173h38m51.578014111s) [noise=true , address=145.79.227.20:4437 , piece_id=G5P6OYXVAAQDZKTPPHDNZLMYTT6L2TFWKH7N4F7L6742BWHOQGBQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.026887915 (173h38m51.578358366s) [noise=true , address=45.91.116.14:4189 , piece_id=NUMS6Q3F2OUAQ2PSKE2CZK2WKRQLZUZPWVBA3UA6UNKDQACPYSJQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.027039624 (173h38m51.578510085s) [noise=true , address=145.79.227.27:4691 , piece_id=NSX5S2D7ZSA5I57B4EAQEQXEVUMPKNMRKRV3RHWGWJ2DN7SWUPGQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.027374482 (173h38m51.578844973s) [noise=true , address=145.79.227.31:4798 , piece_id=DPLALMJC4JPC5D7BOK5SOSDFDJFDDPGSJWP4XDDQE22AMFNAKATQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.027472337 (173h38m51.578942798s) [noise=true , address=45.91.116.7:4010 , piece_id=EO7IEOCWEJAMZZU4BLIB5ALFACWTWZE3QCXU3FUY4FKC4P444ORA, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.027920821 (173h38m51.579391283s) [noise=true , address=145.79.227.23:4547 , piece_id=P4WX2ZDHV2Z6BFMF6IACIYW4DXKYFVSULUTF7EK2EMRO5OA5X5RQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.028194272 (173h38m51.579664743s) [noise=true , address=145.79.227.32:4834 , piece_id=CBK2UQ252EPFQDTOYS2X5WAUTJN4L2DTHGIOJXFZXWBOLGPRETXQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.028430262 (173h38m51.579900723s) [noise=true , address=145.79.227.9:4366 , piece_id=YFWF2YL2YPYVMP5DHJKRUGVUUS2ALG4DRJ3HUZCDZM4N3N676FNQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.029687726 (173h38m51.581158177s) [noise=true , address=145.79.227.20:4438 , piece_id=6HY2HDNJAZ552ZKXFJ7VNTQNBAQU5ZQLNKAG2NZZWZQHN6EEDG6Q, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.031784128 (173h38m51.583254589s) [noise=true , address=45.91.116.5:4298 , piece_id=ARGPAZE6CTMM6UXSMPSVHDBEQWKVGKY7WXOVLPB7UH3JZYMDB7JQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.034100830 (173h38m51.585571291s) [noise=true , address=77.90.31.10:4518 , piece_id=MSKQ66KE4WANAE4AWOD2IKY3PQN2YGZ5N4W3NXGKXSOLE7B6H6WQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:54.850200282 (173h38m53.401670733s) [upload=152907]: Next returned nil segment Nov 16 13:42:54.850219118 (173h38m53.401689569s) [upload=152907]: Waiting for error group managing segments... Nov 16 13:42:55.154333032 (173h38m53.705803493s) [noise=true , address=45.91.116.7:4010 , piece_id=EO7IEOCWEJAMZZU4BLIB5ALFACWTWZE3QCXU3FUY4FKC4P444ORA, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:42:55.201689040 (173h38m53.753159501s) [noise=true , address=77.90.31.10:4518 , piece_id=MSKQ66KE4WANAE4AWOD2IKY3PQN2YGZ5N4W3NXGKXSOLE7B6H6WQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:42:55.232313345 (173h38m53.783783816s) [noise=true , address=45.91.116.5:4298 , piece_id=ARGPAZE6CTMM6UXSMPSVHDBEQWKVGKY7WXOVLPB7UH3JZYMDB7JQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:42:55.260281393 (173h38m53.811751854s) [noise=true , address=45.91.116.6:4974 , piece_id=ZTS3H7E6A3VJECLRRKRKSVX5PH3VSH7F2HAUPQSPH56NCI6RTFBQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:00.229127085 (173h38m58.780597546s) [noise=true , address=145.79.227.20:4438 , piece_id=6HY2HDNJAZ552ZKXFJ7VNTQNBAQU5ZQLNKAG2NZZWZQHN6EEDG6Q, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:00.358241119 (173h38m58.909711581s) [noise=true , address=145.79.227.32:4834 , piece_id=CBK2UQ252EPFQDTOYS2X5WAUTJN4L2DTHGIOJXFZXWBOLGPRETXQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:00.646958215 (173h38m59.198428676s) [noise=true , address=145.79.227.31:4798 , piece_id=DPLALMJC4JPC5D7BOK5SOSDFDJFDDPGSJWP4XDDQE22AMFNAKATQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:00.908824080 (173h38m59.460294541s) [noise=true , address=145.79.227.23:4547 , piece_id=P4WX2ZDHV2Z6BFMF6IACIYW4DXKYFVSULUTF7EK2EMRO5OA5X5RQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:01.155346527 (173h38m59.706816989s) [noise=true , address=45.91.116.14:4189 , piece_id=NUMS6Q3F2OUAQ2PSKE2CZK2WKRQLZUZPWVBA3UA6UNKDQACPYSJQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:01.839444794 (173h39m0.390915255s) [noise=true , address=145.79.227.11:4732 , piece_id=GR6SASHNUTHAEOUPK7DWKVPHRGMG72FQMF4RINJ5NTIQQGFCVZZQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:02.591249579 (173h39m1.14272004s) [noise=true , address=145.79.227.10:4402 , piece_id=DNT5QVAMRO5OMBNVTDRRUKACMM7M5472LD5E4ASKYIBKBNWYAVRA, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:02.616126740 (173h39m1.167597202s) [noise=true , address=145.79.227.31:4797 , piece_id=UKP3MHXBSALCWFIHCTSR5JFFXEWWAWGDS4JYUYS2JEREQQKTPJSA, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:02.734605482 (173h39m1.286075943s) [noise=true , address=145.79.227.27:4691 , piece_id=NSX5S2D7ZSA5I57B4EAQEQXEVUMPKNMRKRV3RHWGWJ2DN7SWUPGQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:02.734638004 (173h39m1.286108475s) [seg_pos={0 0}, upload=152907]: Segment reached optimal threshold of 13 pieces. Nov 16 13:43:02.735049748 (173h39m1.286520209s) [noise=true , address=145.79.227.9:4366 , piece_id=YFWF2YL2YPYVMP5DHJKRUGVUUS2ALG4DRJ3HUZCDZM4N3N676FNQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy): protocol: write tcp 188.166.225.170:60100->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:60100->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60100->145.79.227.9:4366: use of closed network connection Nov 16 13:43:02.735060949 (173h39m1.28653141s) [noise=true , address=145.79.227.9:4366 , piece_id=YFWF2YL2YPYVMP5DHJKRUGVUUS2ALG4DRJ3HUZCDZM4N3N676FNQ, seg_pos={0 0}, upload=152907]: Long tail cancellation due to optimal threshold Nov 16 13:43:02.735509193 (173h39m1.286979654s) [noise=true , address=77.90.31.14:4662 , piece_id=S5PPMCKIKSOCLUJ4CEH7ZDW3TVUZ7GTFFEIPCQK6FX25UIOUQHHQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): protocol: write tcp 188.166.225.170:50196->77.90.31.14:4662: use of closed network connection; write tcp 188.166.225.170:50196->77.90.31.14:4662: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50196->77.90.31.14:4662: use of closed network connection Nov 16 13:43:02.735518792 (173h39m1.286989253s) [noise=true , address=77.90.31.14:4662 , piece_id=S5PPMCKIKSOCLUJ4CEH7ZDW3TVUZ7GTFFEIPCQK6FX25UIOUQHHQ, seg_pos={0 0}, upload=152907]: Long tail cancellation due to optimal threshold Nov 16 13:43:02.736224335 (173h39m1.287694796s) [noise=true , address=145.79.227.20:4437 , piece_id=G5P6OYXVAAQDZKTPPHDNZLMYTT6L2TFWKH7N4F7L6742BWHOQGBQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV): protocol: read tcp 188.166.225.170:35300->145.79.227.20:4437: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 13:43:02.736239203 (173h39m1.287709654s) [noise=true , address=145.79.227.20:4437 , piece_id=G5P6OYXVAAQDZKTPPHDNZLMYTT6L2TFWKH7N4F7L6742BWHOQGBQ, seg_pos={0 0}, upload=152907]: Long tail cancellation due to optimal threshold Nov 16 13:43:02.736270833 (173h39m1.287741294s) [seg_pos={0 0}, upload=152907]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 16 13:43:02.736437060 (173h39m1.287907521s) [upload=152907]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 16 13:43:02.736444273 (173h39m1.287914735s) [upload=152907]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 16 13:43:05.031396865 (173h39m3.582867336s) [upload=152907]: Done uploading segments... Nov 16 13:43:05.031422193 (173h39m3.582892654s) [upload=152907]: Upload finished. err: <nil> Duration: 5.991923288s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 5.962372086s trace: https://jaeger.dev.storj.io/trace/3c77028f820d12f9 FAIL (2025-11-16 14:10:38 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, Piece ID: UKP3MHXBSALCWFIHCTSR5JFFXEWWAWGDS4JYUYS2JEREQQKTPJSA) context deadline exceeded; piecestore: (Node ID: 12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, Piece ID: DNT5QVAMRO5OMBNVTDRRUKACMM7M5472LD5E4ASKYIBKBNWYAVRA) context deadline exceeded; piecestore: (Node ID: 17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, Piece ID: DPLALMJC4JPC5D7BOK5SOSDFDJFDDPGSJWP4XDDQE22AMFNAKATQ) context deadline exceeded ======LOGS====== gment. Inline: false Nov 16 13:42:52.666975419 (173h38m51.21844588s) [upload=152907]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 16 13:42:52.666980539 (173h38m51.21845099s) [upload=152907]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 16 13:42:53.024947049 (173h38m51.5764175s) [seg_pos={0 0}, upload=152907]: Begin upload segment... Nov 16 13:42:53.025007074 (173h38m51.576477535s) [seg_pos={0 0}, upload=152907]: Done begin upload segment. Nov 16 13:42:53.025087707 (173h38m51.576558158s) [noise=true , address=77.90.31.14:4662 , piece_id=S5PPMCKIKSOCLUJ4CEH7ZDW3TVUZ7GTFFEIPCQK6FX25UIOUQHHQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.025098377 (173h38m51.576568828s) [noise=true , address=145.79.227.31:4797 , piece_id=UKP3MHXBSALCWFIHCTSR5JFFXEWWAWGDS4JYUYS2JEREQQKTPJSA, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.025961451 (173h38m51.577431912s) [noise=true , address=45.91.116.6:4974 , piece_id=ZTS3H7E6A3VJECLRRKRKSVX5PH3VSH7F2HAUPQSPH56NCI6RTFBQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.026101888 (173h38m51.577572339s) [noise=true , address=145.79.227.10:4402 , piece_id=DNT5QVAMRO5OMBNVTDRRUKACMM7M5472LD5E4ASKYIBKBNWYAVRA, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.026397721 (173h38m51.577868182s) [noise=true , address=145.79.227.11:4732 , piece_id=GR6SASHNUTHAEOUPK7DWKVPHRGMG72FQMF4RINJ5NTIQQGFCVZZQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.026543650 (173h38m51.578014111s) [noise=true , address=145.79.227.20:4437 , piece_id=G5P6OYXVAAQDZKTPPHDNZLMYTT6L2TFWKH7N4F7L6742BWHOQGBQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.026887915 (173h38m51.578358366s) [noise=true , address=45.91.116.14:4189 , piece_id=NUMS6Q3F2OUAQ2PSKE2CZK2WKRQLZUZPWVBA3UA6UNKDQACPYSJQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.027039624 (173h38m51.578510085s) [noise=true , address=145.79.227.27:4691 , piece_id=NSX5S2D7ZSA5I57B4EAQEQXEVUMPKNMRKRV3RHWGWJ2DN7SWUPGQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.027374482 (173h38m51.578844973s) [noise=true , address=145.79.227.31:4798 , piece_id=DPLALMJC4JPC5D7BOK5SOSDFDJFDDPGSJWP4XDDQE22AMFNAKATQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.027472337 (173h38m51.578942798s) [noise=true , address=45.91.116.7:4010 , piece_id=EO7IEOCWEJAMZZU4BLIB5ALFACWTWZE3QCXU3FUY4FKC4P444ORA, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.027920821 (173h38m51.579391283s) [noise=true , address=145.79.227.23:4547 , piece_id=P4WX2ZDHV2Z6BFMF6IACIYW4DXKYFVSULUTF7EK2EMRO5OA5X5RQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.028194272 (173h38m51.579664743s) [noise=true , address=145.79.227.32:4834 , piece_id=CBK2UQ252EPFQDTOYS2X5WAUTJN4L2DTHGIOJXFZXWBOLGPRETXQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.028430262 (173h38m51.579900723s) [noise=true , address=145.79.227.9:4366 , piece_id=YFWF2YL2YPYVMP5DHJKRUGVUUS2ALG4DRJ3HUZCDZM4N3N676FNQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.029687726 (173h38m51.581158177s) [noise=true , address=145.79.227.20:4438 , piece_id=6HY2HDNJAZ552ZKXFJ7VNTQNBAQU5ZQLNKAG2NZZWZQHN6EEDG6Q, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.031784128 (173h38m51.583254589s) [noise=true , address=45.91.116.5:4298 , piece_id=ARGPAZE6CTMM6UXSMPSVHDBEQWKVGKY7WXOVLPB7UH3JZYMDB7JQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.034100830 (173h38m51.585571291s) [noise=true , address=77.90.31.10:4518 , piece_id=MSKQ66KE4WANAE4AWOD2IKY3PQN2YGZ5N4W3NXGKXSOLE7B6H6WQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:54.850200282 (173h38m53.401670733s) [upload=152907]: Next returned nil segment Nov 16 13:42:54.850219118 (173h38m53.401689569s) [upload=152907]: Waiting for error group managing segments... Nov 16 13:42:55.154333032 (173h38m53.705803493s) [noise=true , address=45.91.116.7:4010 , piece_id=EO7IEOCWEJAMZZU4BLIB5ALFACWTWZE3QCXU3FUY4FKC4P444ORA, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:42:55.201689040 (173h38m53.753159501s) [noise=true , address=77.90.31.10:4518 , piece_id=MSKQ66KE4WANAE4AWOD2IKY3PQN2YGZ5N4W3NXGKXSOLE7B6H6WQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:42:55.232313345 (173h38m53.783783816s) [noise=true , address=45.91.116.5:4298 , piece_id=ARGPAZE6CTMM6UXSMPSVHDBEQWKVGKY7WXOVLPB7UH3JZYMDB7JQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:42:55.260281393 (173h38m53.811751854s) [noise=true , address=45.91.116.6:4974 , piece_id=ZTS3H7E6A3VJECLRRKRKSVX5PH3VSH7F2HAUPQSPH56NCI6RTFBQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:00.229127085 (173h38m58.780597546s) [noise=true , address=145.79.227.20:4438 , piece_id=6HY2HDNJAZ552ZKXFJ7VNTQNBAQU5ZQLNKAG2NZZWZQHN6EEDG6Q, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:00.358241119 (173h38m58.909711581s) [noise=true , address=145.79.227.32:4834 , piece_id=CBK2UQ252EPFQDTOYS2X5WAUTJN4L2DTHGIOJXFZXWBOLGPRETXQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:00.646958215 (173h38m59.198428676s) [noise=true , address=145.79.227.31:4798 , piece_id=DPLALMJC4JPC5D7BOK5SOSDFDJFDDPGSJWP4XDDQE22AMFNAKATQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:00.908824080 (173h38m59.460294541s) [noise=true , address=145.79.227.23:4547 , piece_id=P4WX2ZDHV2Z6BFMF6IACIYW4DXKYFVSULUTF7EK2EMRO5OA5X5RQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:01.155346527 (173h38m59.706816989s) [noise=true , address=45.91.116.14:4189 , piece_id=NUMS6Q3F2OUAQ2PSKE2CZK2WKRQLZUZPWVBA3UA6UNKDQACPYSJQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:01.839444794 (173h39m0.390915255s) [noise=true , address=145.79.227.11:4732 , piece_id=GR6SASHNUTHAEOUPK7DWKVPHRGMG72FQMF4RINJ5NTIQQGFCVZZQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:02.591249579 (173h39m1.14272004s) [noise=true , address=145.79.227.10:4402 , piece_id=DNT5QVAMRO5OMBNVTDRRUKACMM7M5472LD5E4ASKYIBKBNWYAVRA, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:02.616126740 (173h39m1.167597202s) [noise=true , address=145.79.227.31:4797 , piece_id=UKP3MHXBSALCWFIHCTSR5JFFXEWWAWGDS4JYUYS2JEREQQKTPJSA, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:02.734605482 (173h39m1.286075943s) [noise=true , address=145.79.227.27:4691 , piece_id=NSX5S2D7ZSA5I57B4EAQEQXEVUMPKNMRKRV3RHWGWJ2DN7SWUPGQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:02.734638004 (173h39m1.286108475s) [seg_pos={0 0}, upload=152907]: Segment reached optimal threshold of 13 pieces. Nov 16 13:43:02.735049748 (173h39m1.286520209s) [noise=true , address=145.79.227.9:4366 , piece_id=YFWF2YL2YPYVMP5DHJKRUGVUUS2ALG4DRJ3HUZCDZM4N3N676FNQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy): protocol: write tcp 188.166.225.170:60100->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:60100->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60100->145.79.227.9:4366: use of closed network connection Nov 16 13:43:02.735060949 (173h39m1.28653141s) [noise=true , address=145.79.227.9:4366 , piece_id=YFWF2YL2YPYVMP5DHJKRUGVUUS2ALG4DRJ3HUZCDZM4N3N676FNQ, seg_pos={0 0}, upload=152907]: Long tail cancellation due to optimal threshold Nov 16 13:43:02.735509193 (173h39m1.286979654s) [noise=true , address=77.90.31.14:4662 , piece_id=S5PPMCKIKSOCLUJ4CEH7ZDW3TVUZ7GTFFEIPCQK6FX25UIOUQHHQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): protocol: write tcp 188.166.225.170:50196->77.90.31.14:4662: use of closed network connection; write tcp 188.166.225.170:50196->77.90.31.14:4662: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50196->77.90.31.14:4662: use of closed network connection Nov 16 13:43:02.735518792 (173h39m1.286989253s) [noise=true , address=77.90.31.14:4662 , piece_id=S5PPMCKIKSOCLUJ4CEH7ZDW3TVUZ7GTFFEIPCQK6FX25UIOUQHHQ, seg_pos={0 0}, upload=152907]: Long tail cancellation due to optimal threshold Nov 16 13:43:02.736224335 (173h39m1.287694796s) [noise=true , address=145.79.227.20:4437 , piece_id=G5P6OYXVAAQDZKTPPHDNZLMYTT6L2TFWKH7N4F7L6742BWHOQGBQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV): protocol: read tcp 188.166.225.170:35300->145.79.227.20:4437: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 13:43:02.736239203 (173h39m1.287709654s) [noise=true , address=145.79.227.20:4437 , piece_id=G5P6OYXVAAQDZKTPPHDNZLMYTT6L2TFWKH7N4F7L6742BWHOQGBQ, seg_pos={0 0}, upload=152907]: Long tail cancellation due to optimal threshold Nov 16 13:43:02.736270833 (173h39m1.287741294s) [seg_pos={0 0}, upload=152907]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 16 13:43:02.736437060 (173h39m1.287907521s) [upload=152907]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 16 13:43:02.736444273 (173h39m1.287914735s) [upload=152907]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 16 13:43:05.031396865 (173h39m3.582867336s) [upload=152907]: Done uploading segments... Nov 16 13:43:05.031422193 (173h39m3.582892654s) [upload=152907]: Upload finished. err: <nil> Duration: 5.96239994s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 9 out of 60 is failed (threshold=6) [.....XX....X......X..XX....X......................X........X] duration: 5.000900742s trace: https://jaeger.dev.storj.io/trace/771941b4ae81b918 FAIL (2025-11-16 14:10:23 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000981685s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 6.05000897s trace: https://jaeger.dev.storj.io/trace/3c77028f820d12f9 FAIL (2025-11-16 14:09:45 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, Piece ID: UKP3MHXBSALCWFIHCTSR5JFFXEWWAWGDS4JYUYS2JEREQQKTPJSA) context deadline exceeded; piecestore: (Node ID: 12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, Piece ID: DNT5QVAMRO5OMBNVTDRRUKACMM7M5472LD5E4ASKYIBKBNWYAVRA) context deadline exceeded; piecestore: (Node ID: 17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, Piece ID: DPLALMJC4JPC5D7BOK5SOSDFDJFDDPGSJWP4XDDQE22AMFNAKATQ) context deadline exceeded ======LOGS====== gment. Inline: false Nov 16 13:42:52.666975419 (173h38m51.21844588s) [upload=152907]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 16 13:42:52.666980539 (173h38m51.21845099s) [upload=152907]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 16 13:42:53.024947049 (173h38m51.5764175s) [seg_pos={0 0}, upload=152907]: Begin upload segment... Nov 16 13:42:53.025007074 (173h38m51.576477535s) [seg_pos={0 0}, upload=152907]: Done begin upload segment. Nov 16 13:42:53.025087707 (173h38m51.576558158s) [noise=true , address=77.90.31.14:4662 , piece_id=S5PPMCKIKSOCLUJ4CEH7ZDW3TVUZ7GTFFEIPCQK6FX25UIOUQHHQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.025098377 (173h38m51.576568828s) [noise=true , address=145.79.227.31:4797 , piece_id=UKP3MHXBSALCWFIHCTSR5JFFXEWWAWGDS4JYUYS2JEREQQKTPJSA, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.025961451 (173h38m51.577431912s) [noise=true , address=45.91.116.6:4974 , piece_id=ZTS3H7E6A3VJECLRRKRKSVX5PH3VSH7F2HAUPQSPH56NCI6RTFBQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.026101888 (173h38m51.577572339s) [noise=true , address=145.79.227.10:4402 , piece_id=DNT5QVAMRO5OMBNVTDRRUKACMM7M5472LD5E4ASKYIBKBNWYAVRA, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.026397721 (173h38m51.577868182s) [noise=true , address=145.79.227.11:4732 , piece_id=GR6SASHNUTHAEOUPK7DWKVPHRGMG72FQMF4RINJ5NTIQQGFCVZZQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.026543650 (173h38m51.578014111s) [noise=true , address=145.79.227.20:4437 , piece_id=G5P6OYXVAAQDZKTPPHDNZLMYTT6L2TFWKH7N4F7L6742BWHOQGBQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.026887915 (173h38m51.578358366s) [noise=true , address=45.91.116.14:4189 , piece_id=NUMS6Q3F2OUAQ2PSKE2CZK2WKRQLZUZPWVBA3UA6UNKDQACPYSJQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.027039624 (173h38m51.578510085s) [noise=true , address=145.79.227.27:4691 , piece_id=NSX5S2D7ZSA5I57B4EAQEQXEVUMPKNMRKRV3RHWGWJ2DN7SWUPGQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.027374482 (173h38m51.578844973s) [noise=true , address=145.79.227.31:4798 , piece_id=DPLALMJC4JPC5D7BOK5SOSDFDJFDDPGSJWP4XDDQE22AMFNAKATQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.027472337 (173h38m51.578942798s) [noise=true , address=45.91.116.7:4010 , piece_id=EO7IEOCWEJAMZZU4BLIB5ALFACWTWZE3QCXU3FUY4FKC4P444ORA, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.027920821 (173h38m51.579391283s) [noise=true , address=145.79.227.23:4547 , piece_id=P4WX2ZDHV2Z6BFMF6IACIYW4DXKYFVSULUTF7EK2EMRO5OA5X5RQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.028194272 (173h38m51.579664743s) [noise=true , address=145.79.227.32:4834 , piece_id=CBK2UQ252EPFQDTOYS2X5WAUTJN4L2DTHGIOJXFZXWBOLGPRETXQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.028430262 (173h38m51.579900723s) [noise=true , address=145.79.227.9:4366 , piece_id=YFWF2YL2YPYVMP5DHJKRUGVUUS2ALG4DRJ3HUZCDZM4N3N676FNQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.029687726 (173h38m51.581158177s) [noise=true , address=145.79.227.20:4438 , piece_id=6HY2HDNJAZ552ZKXFJ7VNTQNBAQU5ZQLNKAG2NZZWZQHN6EEDG6Q, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.031784128 (173h38m51.583254589s) [noise=true , address=45.91.116.5:4298 , piece_id=ARGPAZE6CTMM6UXSMPSVHDBEQWKVGKY7WXOVLPB7UH3JZYMDB7JQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.034100830 (173h38m51.585571291s) [noise=true , address=77.90.31.10:4518 , piece_id=MSKQ66KE4WANAE4AWOD2IKY3PQN2YGZ5N4W3NXGKXSOLE7B6H6WQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:54.850200282 (173h38m53.401670733s) [upload=152907]: Next returned nil segment Nov 16 13:42:54.850219118 (173h38m53.401689569s) [upload=152907]: Waiting for error group managing segments... Nov 16 13:42:55.154333032 (173h38m53.705803493s) [noise=true , address=45.91.116.7:4010 , piece_id=EO7IEOCWEJAMZZU4BLIB5ALFACWTWZE3QCXU3FUY4FKC4P444ORA, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:42:55.201689040 (173h38m53.753159501s) [noise=true , address=77.90.31.10:4518 , piece_id=MSKQ66KE4WANAE4AWOD2IKY3PQN2YGZ5N4W3NXGKXSOLE7B6H6WQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:42:55.232313345 (173h38m53.783783816s) [noise=true , address=45.91.116.5:4298 , piece_id=ARGPAZE6CTMM6UXSMPSVHDBEQWKVGKY7WXOVLPB7UH3JZYMDB7JQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:42:55.260281393 (173h38m53.811751854s) [noise=true , address=45.91.116.6:4974 , piece_id=ZTS3H7E6A3VJECLRRKRKSVX5PH3VSH7F2HAUPQSPH56NCI6RTFBQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:00.229127085 (173h38m58.780597546s) [noise=true , address=145.79.227.20:4438 , piece_id=6HY2HDNJAZ552ZKXFJ7VNTQNBAQU5ZQLNKAG2NZZWZQHN6EEDG6Q, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:00.358241119 (173h38m58.909711581s) [noise=true , address=145.79.227.32:4834 , piece_id=CBK2UQ252EPFQDTOYS2X5WAUTJN4L2DTHGIOJXFZXWBOLGPRETXQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:00.646958215 (173h38m59.198428676s) [noise=true , address=145.79.227.31:4798 , piece_id=DPLALMJC4JPC5D7BOK5SOSDFDJFDDPGSJWP4XDDQE22AMFNAKATQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:00.908824080 (173h38m59.460294541s) [noise=true , address=145.79.227.23:4547 , piece_id=P4WX2ZDHV2Z6BFMF6IACIYW4DXKYFVSULUTF7EK2EMRO5OA5X5RQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:01.155346527 (173h38m59.706816989s) [noise=true , address=45.91.116.14:4189 , piece_id=NUMS6Q3F2OUAQ2PSKE2CZK2WKRQLZUZPWVBA3UA6UNKDQACPYSJQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:01.839444794 (173h39m0.390915255s) [noise=true , address=145.79.227.11:4732 , piece_id=GR6SASHNUTHAEOUPK7DWKVPHRGMG72FQMF4RINJ5NTIQQGFCVZZQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:02.591249579 (173h39m1.14272004s) [noise=true , address=145.79.227.10:4402 , piece_id=DNT5QVAMRO5OMBNVTDRRUKACMM7M5472LD5E4ASKYIBKBNWYAVRA, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:02.616126740 (173h39m1.167597202s) [noise=true , address=145.79.227.31:4797 , piece_id=UKP3MHXBSALCWFIHCTSR5JFFXEWWAWGDS4JYUYS2JEREQQKTPJSA, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:02.734605482 (173h39m1.286075943s) [noise=true , address=145.79.227.27:4691 , piece_id=NSX5S2D7ZSA5I57B4EAQEQXEVUMPKNMRKRV3RHWGWJ2DN7SWUPGQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:02.734638004 (173h39m1.286108475s) [seg_pos={0 0}, upload=152907]: Segment reached optimal threshold of 13 pieces. Nov 16 13:43:02.735049748 (173h39m1.286520209s) [noise=true , address=145.79.227.9:4366 , piece_id=YFWF2YL2YPYVMP5DHJKRUGVUUS2ALG4DRJ3HUZCDZM4N3N676FNQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy): protocol: write tcp 188.166.225.170:60100->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:60100->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60100->145.79.227.9:4366: use of closed network connection Nov 16 13:43:02.735060949 (173h39m1.28653141s) [noise=true , address=145.79.227.9:4366 , piece_id=YFWF2YL2YPYVMP5DHJKRUGVUUS2ALG4DRJ3HUZCDZM4N3N676FNQ, seg_pos={0 0}, upload=152907]: Long tail cancellation due to optimal threshold Nov 16 13:43:02.735509193 (173h39m1.286979654s) [noise=true , address=77.90.31.14:4662 , piece_id=S5PPMCKIKSOCLUJ4CEH7ZDW3TVUZ7GTFFEIPCQK6FX25UIOUQHHQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): protocol: write tcp 188.166.225.170:50196->77.90.31.14:4662: use of closed network connection; write tcp 188.166.225.170:50196->77.90.31.14:4662: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50196->77.90.31.14:4662: use of closed network connection Nov 16 13:43:02.735518792 (173h39m1.286989253s) [noise=true , address=77.90.31.14:4662 , piece_id=S5PPMCKIKSOCLUJ4CEH7ZDW3TVUZ7GTFFEIPCQK6FX25UIOUQHHQ, seg_pos={0 0}, upload=152907]: Long tail cancellation due to optimal threshold Nov 16 13:43:02.736224335 (173h39m1.287694796s) [noise=true , address=145.79.227.20:4437 , piece_id=G5P6OYXVAAQDZKTPPHDNZLMYTT6L2TFWKH7N4F7L6742BWHOQGBQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV): protocol: read tcp 188.166.225.170:35300->145.79.227.20:4437: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 13:43:02.736239203 (173h39m1.287709654s) [noise=true , address=145.79.227.20:4437 , piece_id=G5P6OYXVAAQDZKTPPHDNZLMYTT6L2TFWKH7N4F7L6742BWHOQGBQ, seg_pos={0 0}, upload=152907]: Long tail cancellation due to optimal threshold Nov 16 13:43:02.736270833 (173h39m1.287741294s) [seg_pos={0 0}, upload=152907]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 16 13:43:02.736437060 (173h39m1.287907521s) [upload=152907]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 16 13:43:02.736444273 (173h39m1.287914735s) [upload=152907]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 16 13:43:05.031396865 (173h39m3.582867336s) [upload=152907]: Done uploading segments... Nov 16 13:43:05.031422193 (173h39m3.582892654s) [upload=152907]: Upload finished. err: <nil> Duration: 6.050039046s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 8 out of 60 is failed (threshold=6) [......XX....X......X..XX....X......................X........] duration: 3.524123513s trace: https://jaeger.dev.storj.io/trace/7ba24b09fac1d319 FAIL (2025-11-16 14:09:11 UTC) orbiter warning: context deadline exceeded Duration: 3.524173038s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 12.29978001s trace: https://jaeger.dev.storj.io/trace/129f27707330b16c FAIL (2025-11-16 14:09:03 UTC) orbiter warning: context deadline exceeded Duration: 12.299805959s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 8 out of 60 is failed (threshold=6) [.......XX....X......X..XX....X......................X.......] duration: 3.584304653s trace: https://jaeger.dev.storj.io/trace/7ba24b09fac1d319 FAIL (2025-11-16 14:08:46 UTC) orbiter warning: context deadline exceeded Duration: 3.584403812s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 5.898489902s trace: https://jaeger.dev.storj.io/trace/3c77028f820d12f9 FAIL (2025-11-16 14:08:36 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, Piece ID: UKP3MHXBSALCWFIHCTSR5JFFXEWWAWGDS4JYUYS2JEREQQKTPJSA) context deadline exceeded; piecestore: (Node ID: 12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, Piece ID: DNT5QVAMRO5OMBNVTDRRUKACMM7M5472LD5E4ASKYIBKBNWYAVRA) context deadline exceeded; piecestore: (Node ID: 17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, Piece ID: DPLALMJC4JPC5D7BOK5SOSDFDJFDDPGSJWP4XDDQE22AMFNAKATQ) context deadline exceeded ======LOGS====== gment. Inline: false Nov 16 13:42:52.666975419 (173h38m51.21844588s) [upload=152907]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 16 13:42:52.666980539 (173h38m51.21845099s) [upload=152907]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 16 13:42:53.024947049 (173h38m51.5764175s) [seg_pos={0 0}, upload=152907]: Begin upload segment... Nov 16 13:42:53.025007074 (173h38m51.576477535s) [seg_pos={0 0}, upload=152907]: Done begin upload segment. Nov 16 13:42:53.025087707 (173h38m51.576558158s) [noise=true , address=77.90.31.14:4662 , piece_id=S5PPMCKIKSOCLUJ4CEH7ZDW3TVUZ7GTFFEIPCQK6FX25UIOUQHHQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.025098377 (173h38m51.576568828s) [noise=true , address=145.79.227.31:4797 , piece_id=UKP3MHXBSALCWFIHCTSR5JFFXEWWAWGDS4JYUYS2JEREQQKTPJSA, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.025961451 (173h38m51.577431912s) [noise=true , address=45.91.116.6:4974 , piece_id=ZTS3H7E6A3VJECLRRKRKSVX5PH3VSH7F2HAUPQSPH56NCI6RTFBQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.026101888 (173h38m51.577572339s) [noise=true , address=145.79.227.10:4402 , piece_id=DNT5QVAMRO5OMBNVTDRRUKACMM7M5472LD5E4ASKYIBKBNWYAVRA, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.026397721 (173h38m51.577868182s) [noise=true , address=145.79.227.11:4732 , piece_id=GR6SASHNUTHAEOUPK7DWKVPHRGMG72FQMF4RINJ5NTIQQGFCVZZQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.026543650 (173h38m51.578014111s) [noise=true , address=145.79.227.20:4437 , piece_id=G5P6OYXVAAQDZKTPPHDNZLMYTT6L2TFWKH7N4F7L6742BWHOQGBQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.026887915 (173h38m51.578358366s) [noise=true , address=45.91.116.14:4189 , piece_id=NUMS6Q3F2OUAQ2PSKE2CZK2WKRQLZUZPWVBA3UA6UNKDQACPYSJQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.027039624 (173h38m51.578510085s) [noise=true , address=145.79.227.27:4691 , piece_id=NSX5S2D7ZSA5I57B4EAQEQXEVUMPKNMRKRV3RHWGWJ2DN7SWUPGQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.027374482 (173h38m51.578844973s) [noise=true , address=145.79.227.31:4798 , piece_id=DPLALMJC4JPC5D7BOK5SOSDFDJFDDPGSJWP4XDDQE22AMFNAKATQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.027472337 (173h38m51.578942798s) [noise=true , address=45.91.116.7:4010 , piece_id=EO7IEOCWEJAMZZU4BLIB5ALFACWTWZE3QCXU3FUY4FKC4P444ORA, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.027920821 (173h38m51.579391283s) [noise=true , address=145.79.227.23:4547 , piece_id=P4WX2ZDHV2Z6BFMF6IACIYW4DXKYFVSULUTF7EK2EMRO5OA5X5RQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.028194272 (173h38m51.579664743s) [noise=true , address=145.79.227.32:4834 , piece_id=CBK2UQ252EPFQDTOYS2X5WAUTJN4L2DTHGIOJXFZXWBOLGPRETXQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.028430262 (173h38m51.579900723s) [noise=true , address=145.79.227.9:4366 , piece_id=YFWF2YL2YPYVMP5DHJKRUGVUUS2ALG4DRJ3HUZCDZM4N3N676FNQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.029687726 (173h38m51.581158177s) [noise=true , address=145.79.227.20:4438 , piece_id=6HY2HDNJAZ552ZKXFJ7VNTQNBAQU5ZQLNKAG2NZZWZQHN6EEDG6Q, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.031784128 (173h38m51.583254589s) [noise=true , address=45.91.116.5:4298 , piece_id=ARGPAZE6CTMM6UXSMPSVHDBEQWKVGKY7WXOVLPB7UH3JZYMDB7JQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.034100830 (173h38m51.585571291s) [noise=true , address=77.90.31.10:4518 , piece_id=MSKQ66KE4WANAE4AWOD2IKY3PQN2YGZ5N4W3NXGKXSOLE7B6H6WQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:54.850200282 (173h38m53.401670733s) [upload=152907]: Next returned nil segment Nov 16 13:42:54.850219118 (173h38m53.401689569s) [upload=152907]: Waiting for error group managing segments... Nov 16 13:42:55.154333032 (173h38m53.705803493s) [noise=true , address=45.91.116.7:4010 , piece_id=EO7IEOCWEJAMZZU4BLIB5ALFACWTWZE3QCXU3FUY4FKC4P444ORA, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:42:55.201689040 (173h38m53.753159501s) [noise=true , address=77.90.31.10:4518 , piece_id=MSKQ66KE4WANAE4AWOD2IKY3PQN2YGZ5N4W3NXGKXSOLE7B6H6WQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:42:55.232313345 (173h38m53.783783816s) [noise=true , address=45.91.116.5:4298 , piece_id=ARGPAZE6CTMM6UXSMPSVHDBEQWKVGKY7WXOVLPB7UH3JZYMDB7JQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:42:55.260281393 (173h38m53.811751854s) [noise=true , address=45.91.116.6:4974 , piece_id=ZTS3H7E6A3VJECLRRKRKSVX5PH3VSH7F2HAUPQSPH56NCI6RTFBQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:00.229127085 (173h38m58.780597546s) [noise=true , address=145.79.227.20:4438 , piece_id=6HY2HDNJAZ552ZKXFJ7VNTQNBAQU5ZQLNKAG2NZZWZQHN6EEDG6Q, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:00.358241119 (173h38m58.909711581s) [noise=true , address=145.79.227.32:4834 , piece_id=CBK2UQ252EPFQDTOYS2X5WAUTJN4L2DTHGIOJXFZXWBOLGPRETXQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:00.646958215 (173h38m59.198428676s) [noise=true , address=145.79.227.31:4798 , piece_id=DPLALMJC4JPC5D7BOK5SOSDFDJFDDPGSJWP4XDDQE22AMFNAKATQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:00.908824080 (173h38m59.460294541s) [noise=true , address=145.79.227.23:4547 , piece_id=P4WX2ZDHV2Z6BFMF6IACIYW4DXKYFVSULUTF7EK2EMRO5OA5X5RQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:01.155346527 (173h38m59.706816989s) [noise=true , address=45.91.116.14:4189 , piece_id=NUMS6Q3F2OUAQ2PSKE2CZK2WKRQLZUZPWVBA3UA6UNKDQACPYSJQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:01.839444794 (173h39m0.390915255s) [noise=true , address=145.79.227.11:4732 , piece_id=GR6SASHNUTHAEOUPK7DWKVPHRGMG72FQMF4RINJ5NTIQQGFCVZZQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:02.591249579 (173h39m1.14272004s) [noise=true , address=145.79.227.10:4402 , piece_id=DNT5QVAMRO5OMBNVTDRRUKACMM7M5472LD5E4ASKYIBKBNWYAVRA, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:02.616126740 (173h39m1.167597202s) [noise=true , address=145.79.227.31:4797 , piece_id=UKP3MHXBSALCWFIHCTSR5JFFXEWWAWGDS4JYUYS2JEREQQKTPJSA, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:02.734605482 (173h39m1.286075943s) [noise=true , address=145.79.227.27:4691 , piece_id=NSX5S2D7ZSA5I57B4EAQEQXEVUMPKNMRKRV3RHWGWJ2DN7SWUPGQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:02.734638004 (173h39m1.286108475s) [seg_pos={0 0}, upload=152907]: Segment reached optimal threshold of 13 pieces. Nov 16 13:43:02.735049748 (173h39m1.286520209s) [noise=true , address=145.79.227.9:4366 , piece_id=YFWF2YL2YPYVMP5DHJKRUGVUUS2ALG4DRJ3HUZCDZM4N3N676FNQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy): protocol: write tcp 188.166.225.170:60100->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:60100->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60100->145.79.227.9:4366: use of closed network connection Nov 16 13:43:02.735060949 (173h39m1.28653141s) [noise=true , address=145.79.227.9:4366 , piece_id=YFWF2YL2YPYVMP5DHJKRUGVUUS2ALG4DRJ3HUZCDZM4N3N676FNQ, seg_pos={0 0}, upload=152907]: Long tail cancellation due to optimal threshold Nov 16 13:43:02.735509193 (173h39m1.286979654s) [noise=true , address=77.90.31.14:4662 , piece_id=S5PPMCKIKSOCLUJ4CEH7ZDW3TVUZ7GTFFEIPCQK6FX25UIOUQHHQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): protocol: write tcp 188.166.225.170:50196->77.90.31.14:4662: use of closed network connection; write tcp 188.166.225.170:50196->77.90.31.14:4662: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50196->77.90.31.14:4662: use of closed network connection Nov 16 13:43:02.735518792 (173h39m1.286989253s) [noise=true , address=77.90.31.14:4662 , piece_id=S5PPMCKIKSOCLUJ4CEH7ZDW3TVUZ7GTFFEIPCQK6FX25UIOUQHHQ, seg_pos={0 0}, upload=152907]: Long tail cancellation due to optimal threshold Nov 16 13:43:02.736224335 (173h39m1.287694796s) [noise=true , address=145.79.227.20:4437 , piece_id=G5P6OYXVAAQDZKTPPHDNZLMYTT6L2TFWKH7N4F7L6742BWHOQGBQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV): protocol: read tcp 188.166.225.170:35300->145.79.227.20:4437: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 13:43:02.736239203 (173h39m1.287709654s) [noise=true , address=145.79.227.20:4437 , piece_id=G5P6OYXVAAQDZKTPPHDNZLMYTT6L2TFWKH7N4F7L6742BWHOQGBQ, seg_pos={0 0}, upload=152907]: Long tail cancellation due to optimal threshold Nov 16 13:43:02.736270833 (173h39m1.287741294s) [seg_pos={0 0}, upload=152907]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 16 13:43:02.736437060 (173h39m1.287907521s) [upload=152907]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 16 13:43:02.736444273 (173h39m1.287914735s) [upload=152907]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 16 13:43:05.031396865 (173h39m3.582867336s) [upload=152907]: Done uploading segments... Nov 16 13:43:05.031422193 (173h39m3.582892654s) [upload=152907]: Upload finished. err: <nil> Duration: 5.898582148s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 13.279141093s trace: https://jaeger.dev.storj.io/trace/129f27707330b16c FAIL (2025-11-16 14:08:25 UTC) orbiter warning: context deadline exceeded Duration: 13.279175017s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 8 out of 60 is failed (threshold=6) [........XX....X......X..XX....X......................X......] duration: 3.554363029s trace: https://jaeger.dev.storj.io/trace/7ba24b09fac1d319 FAIL (2025-11-16 14:08:03 UTC) orbiter warning: context deadline exceeded Duration: 3.554396002s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)