Orbiter History

checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 17.879044483s trace: https://jaeger.dev.storj.io/trace/129f27707330b16c FAIL (2025-11-16 13:50:23 UTC) orbiter warning: context deadline exceeded Duration: 17.87907953s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 6.346485116s trace: https://jaeger.dev.storj.io/trace/3c77028f820d12f9 FAIL (2025-11-16 13:50:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, Piece ID: UKP3MHXBSALCWFIHCTSR5JFFXEWWAWGDS4JYUYS2JEREQQKTPJSA) context deadline exceeded; piecestore: (Node ID: 12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, Piece ID: DNT5QVAMRO5OMBNVTDRRUKACMM7M5472LD5E4ASKYIBKBNWYAVRA) context deadline exceeded; piecestore: (Node ID: 17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, Piece ID: DPLALMJC4JPC5D7BOK5SOSDFDJFDDPGSJWP4XDDQE22AMFNAKATQ) context deadline exceeded ======LOGS====== gment. Inline: false Nov 16 13:42:52.666975419 (173h38m51.21844588s) [upload=152907]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 16 13:42:52.666980539 (173h38m51.21845099s) [upload=152907]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 16 13:42:53.024947049 (173h38m51.5764175s) [seg_pos={0 0}, upload=152907]: Begin upload segment... Nov 16 13:42:53.025007074 (173h38m51.576477535s) [seg_pos={0 0}, upload=152907]: Done begin upload segment. Nov 16 13:42:53.025087707 (173h38m51.576558158s) [noise=true , address=77.90.31.14:4662 , piece_id=S5PPMCKIKSOCLUJ4CEH7ZDW3TVUZ7GTFFEIPCQK6FX25UIOUQHHQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.025098377 (173h38m51.576568828s) [noise=true , address=145.79.227.31:4797 , piece_id=UKP3MHXBSALCWFIHCTSR5JFFXEWWAWGDS4JYUYS2JEREQQKTPJSA, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.025961451 (173h38m51.577431912s) [noise=true , address=45.91.116.6:4974 , piece_id=ZTS3H7E6A3VJECLRRKRKSVX5PH3VSH7F2HAUPQSPH56NCI6RTFBQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.026101888 (173h38m51.577572339s) [noise=true , address=145.79.227.10:4402 , piece_id=DNT5QVAMRO5OMBNVTDRRUKACMM7M5472LD5E4ASKYIBKBNWYAVRA, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.026397721 (173h38m51.577868182s) [noise=true , address=145.79.227.11:4732 , piece_id=GR6SASHNUTHAEOUPK7DWKVPHRGMG72FQMF4RINJ5NTIQQGFCVZZQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.026543650 (173h38m51.578014111s) [noise=true , address=145.79.227.20:4437 , piece_id=G5P6OYXVAAQDZKTPPHDNZLMYTT6L2TFWKH7N4F7L6742BWHOQGBQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.026887915 (173h38m51.578358366s) [noise=true , address=45.91.116.14:4189 , piece_id=NUMS6Q3F2OUAQ2PSKE2CZK2WKRQLZUZPWVBA3UA6UNKDQACPYSJQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.027039624 (173h38m51.578510085s) [noise=true , address=145.79.227.27:4691 , piece_id=NSX5S2D7ZSA5I57B4EAQEQXEVUMPKNMRKRV3RHWGWJ2DN7SWUPGQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.027374482 (173h38m51.578844973s) [noise=true , address=145.79.227.31:4798 , piece_id=DPLALMJC4JPC5D7BOK5SOSDFDJFDDPGSJWP4XDDQE22AMFNAKATQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.027472337 (173h38m51.578942798s) [noise=true , address=45.91.116.7:4010 , piece_id=EO7IEOCWEJAMZZU4BLIB5ALFACWTWZE3QCXU3FUY4FKC4P444ORA, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.027920821 (173h38m51.579391283s) [noise=true , address=145.79.227.23:4547 , piece_id=P4WX2ZDHV2Z6BFMF6IACIYW4DXKYFVSULUTF7EK2EMRO5OA5X5RQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.028194272 (173h38m51.579664743s) [noise=true , address=145.79.227.32:4834 , piece_id=CBK2UQ252EPFQDTOYS2X5WAUTJN4L2DTHGIOJXFZXWBOLGPRETXQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.028430262 (173h38m51.579900723s) [noise=true , address=145.79.227.9:4366 , piece_id=YFWF2YL2YPYVMP5DHJKRUGVUUS2ALG4DRJ3HUZCDZM4N3N676FNQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.029687726 (173h38m51.581158177s) [noise=true , address=145.79.227.20:4438 , piece_id=6HY2HDNJAZ552ZKXFJ7VNTQNBAQU5ZQLNKAG2NZZWZQHN6EEDG6Q, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.031784128 (173h38m51.583254589s) [noise=true , address=45.91.116.5:4298 , piece_id=ARGPAZE6CTMM6UXSMPSVHDBEQWKVGKY7WXOVLPB7UH3JZYMDB7JQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.034100830 (173h38m51.585571291s) [noise=true , address=77.90.31.10:4518 , piece_id=MSKQ66KE4WANAE4AWOD2IKY3PQN2YGZ5N4W3NXGKXSOLE7B6H6WQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:54.850200282 (173h38m53.401670733s) [upload=152907]: Next returned nil segment Nov 16 13:42:54.850219118 (173h38m53.401689569s) [upload=152907]: Waiting for error group managing segments... Nov 16 13:42:55.154333032 (173h38m53.705803493s) [noise=true , address=45.91.116.7:4010 , piece_id=EO7IEOCWEJAMZZU4BLIB5ALFACWTWZE3QCXU3FUY4FKC4P444ORA, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:42:55.201689040 (173h38m53.753159501s) [noise=true , address=77.90.31.10:4518 , piece_id=MSKQ66KE4WANAE4AWOD2IKY3PQN2YGZ5N4W3NXGKXSOLE7B6H6WQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:42:55.232313345 (173h38m53.783783816s) [noise=true , address=45.91.116.5:4298 , piece_id=ARGPAZE6CTMM6UXSMPSVHDBEQWKVGKY7WXOVLPB7UH3JZYMDB7JQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:42:55.260281393 (173h38m53.811751854s) [noise=true , address=45.91.116.6:4974 , piece_id=ZTS3H7E6A3VJECLRRKRKSVX5PH3VSH7F2HAUPQSPH56NCI6RTFBQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:00.229127085 (173h38m58.780597546s) [noise=true , address=145.79.227.20:4438 , piece_id=6HY2HDNJAZ552ZKXFJ7VNTQNBAQU5ZQLNKAG2NZZWZQHN6EEDG6Q, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:00.358241119 (173h38m58.909711581s) [noise=true , address=145.79.227.32:4834 , piece_id=CBK2UQ252EPFQDTOYS2X5WAUTJN4L2DTHGIOJXFZXWBOLGPRETXQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:00.646958215 (173h38m59.198428676s) [noise=true , address=145.79.227.31:4798 , piece_id=DPLALMJC4JPC5D7BOK5SOSDFDJFDDPGSJWP4XDDQE22AMFNAKATQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:00.908824080 (173h38m59.460294541s) [noise=true , address=145.79.227.23:4547 , piece_id=P4WX2ZDHV2Z6BFMF6IACIYW4DXKYFVSULUTF7EK2EMRO5OA5X5RQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:01.155346527 (173h38m59.706816989s) [noise=true , address=45.91.116.14:4189 , piece_id=NUMS6Q3F2OUAQ2PSKE2CZK2WKRQLZUZPWVBA3UA6UNKDQACPYSJQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:01.839444794 (173h39m0.390915255s) [noise=true , address=145.79.227.11:4732 , piece_id=GR6SASHNUTHAEOUPK7DWKVPHRGMG72FQMF4RINJ5NTIQQGFCVZZQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:02.591249579 (173h39m1.14272004s) [noise=true , address=145.79.227.10:4402 , piece_id=DNT5QVAMRO5OMBNVTDRRUKACMM7M5472LD5E4ASKYIBKBNWYAVRA, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:02.616126740 (173h39m1.167597202s) [noise=true , address=145.79.227.31:4797 , piece_id=UKP3MHXBSALCWFIHCTSR5JFFXEWWAWGDS4JYUYS2JEREQQKTPJSA, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:02.734605482 (173h39m1.286075943s) [noise=true , address=145.79.227.27:4691 , piece_id=NSX5S2D7ZSA5I57B4EAQEQXEVUMPKNMRKRV3RHWGWJ2DN7SWUPGQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:02.734638004 (173h39m1.286108475s) [seg_pos={0 0}, upload=152907]: Segment reached optimal threshold of 13 pieces. Nov 16 13:43:02.735049748 (173h39m1.286520209s) [noise=true , address=145.79.227.9:4366 , piece_id=YFWF2YL2YPYVMP5DHJKRUGVUUS2ALG4DRJ3HUZCDZM4N3N676FNQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy): protocol: write tcp 188.166.225.170:60100->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:60100->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60100->145.79.227.9:4366: use of closed network connection Nov 16 13:43:02.735060949 (173h39m1.28653141s) [noise=true , address=145.79.227.9:4366 , piece_id=YFWF2YL2YPYVMP5DHJKRUGVUUS2ALG4DRJ3HUZCDZM4N3N676FNQ, seg_pos={0 0}, upload=152907]: Long tail cancellation due to optimal threshold Nov 16 13:43:02.735509193 (173h39m1.286979654s) [noise=true , address=77.90.31.14:4662 , piece_id=S5PPMCKIKSOCLUJ4CEH7ZDW3TVUZ7GTFFEIPCQK6FX25UIOUQHHQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): protocol: write tcp 188.166.225.170:50196->77.90.31.14:4662: use of closed network connection; write tcp 188.166.225.170:50196->77.90.31.14:4662: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50196->77.90.31.14:4662: use of closed network connection Nov 16 13:43:02.735518792 (173h39m1.286989253s) [noise=true , address=77.90.31.14:4662 , piece_id=S5PPMCKIKSOCLUJ4CEH7ZDW3TVUZ7GTFFEIPCQK6FX25UIOUQHHQ, seg_pos={0 0}, upload=152907]: Long tail cancellation due to optimal threshold Nov 16 13:43:02.736224335 (173h39m1.287694796s) [noise=true , address=145.79.227.20:4437 , piece_id=G5P6OYXVAAQDZKTPPHDNZLMYTT6L2TFWKH7N4F7L6742BWHOQGBQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV): protocol: read tcp 188.166.225.170:35300->145.79.227.20:4437: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 13:43:02.736239203 (173h39m1.287709654s) [noise=true , address=145.79.227.20:4437 , piece_id=G5P6OYXVAAQDZKTPPHDNZLMYTT6L2TFWKH7N4F7L6742BWHOQGBQ, seg_pos={0 0}, upload=152907]: Long tail cancellation due to optimal threshold Nov 16 13:43:02.736270833 (173h39m1.287741294s) [seg_pos={0 0}, upload=152907]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 16 13:43:02.736437060 (173h39m1.287907521s) [upload=152907]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 16 13:43:02.736444273 (173h39m1.287914735s) [upload=152907]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 16 13:43:05.031396865 (173h39m3.582867336s) [upload=152907]: Done uploading segments... Nov 16 13:43:05.031422193 (173h39m3.582892654s) [upload=152907]: Upload finished. err: <nil> Duration: 6.346510584s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 11 out of 60 is failed (threshold=6) [..........XX....X........X.........XX....X......X..XX....X..] duration: 5.002068707s trace: https://jaeger.dev.storj.io/trace/69ae5765386935f3 FAIL (2025-11-16 13:49:59 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.002094236s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 6.08105854s trace: https://jaeger.dev.storj.io/trace/3c77028f820d12f9 FAIL (2025-11-16 13:49:45 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, Piece ID: UKP3MHXBSALCWFIHCTSR5JFFXEWWAWGDS4JYUYS2JEREQQKTPJSA) context deadline exceeded; piecestore: (Node ID: 12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, Piece ID: DNT5QVAMRO5OMBNVTDRRUKACMM7M5472LD5E4ASKYIBKBNWYAVRA) context deadline exceeded; piecestore: (Node ID: 17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, Piece ID: DPLALMJC4JPC5D7BOK5SOSDFDJFDDPGSJWP4XDDQE22AMFNAKATQ) context deadline exceeded ======LOGS====== gment. Inline: false Nov 16 13:42:52.666975419 (173h38m51.21844588s) [upload=152907]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 16 13:42:52.666980539 (173h38m51.21845099s) [upload=152907]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 16 13:42:53.024947049 (173h38m51.5764175s) [seg_pos={0 0}, upload=152907]: Begin upload segment... Nov 16 13:42:53.025007074 (173h38m51.576477535s) [seg_pos={0 0}, upload=152907]: Done begin upload segment. Nov 16 13:42:53.025087707 (173h38m51.576558158s) [noise=true , address=77.90.31.14:4662 , piece_id=S5PPMCKIKSOCLUJ4CEH7ZDW3TVUZ7GTFFEIPCQK6FX25UIOUQHHQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.025098377 (173h38m51.576568828s) [noise=true , address=145.79.227.31:4797 , piece_id=UKP3MHXBSALCWFIHCTSR5JFFXEWWAWGDS4JYUYS2JEREQQKTPJSA, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.025961451 (173h38m51.577431912s) [noise=true , address=45.91.116.6:4974 , piece_id=ZTS3H7E6A3VJECLRRKRKSVX5PH3VSH7F2HAUPQSPH56NCI6RTFBQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.026101888 (173h38m51.577572339s) [noise=true , address=145.79.227.10:4402 , piece_id=DNT5QVAMRO5OMBNVTDRRUKACMM7M5472LD5E4ASKYIBKBNWYAVRA, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.026397721 (173h38m51.577868182s) [noise=true , address=145.79.227.11:4732 , piece_id=GR6SASHNUTHAEOUPK7DWKVPHRGMG72FQMF4RINJ5NTIQQGFCVZZQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.026543650 (173h38m51.578014111s) [noise=true , address=145.79.227.20:4437 , piece_id=G5P6OYXVAAQDZKTPPHDNZLMYTT6L2TFWKH7N4F7L6742BWHOQGBQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.026887915 (173h38m51.578358366s) [noise=true , address=45.91.116.14:4189 , piece_id=NUMS6Q3F2OUAQ2PSKE2CZK2WKRQLZUZPWVBA3UA6UNKDQACPYSJQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.027039624 (173h38m51.578510085s) [noise=true , address=145.79.227.27:4691 , piece_id=NSX5S2D7ZSA5I57B4EAQEQXEVUMPKNMRKRV3RHWGWJ2DN7SWUPGQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.027374482 (173h38m51.578844973s) [noise=true , address=145.79.227.31:4798 , piece_id=DPLALMJC4JPC5D7BOK5SOSDFDJFDDPGSJWP4XDDQE22AMFNAKATQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.027472337 (173h38m51.578942798s) [noise=true , address=45.91.116.7:4010 , piece_id=EO7IEOCWEJAMZZU4BLIB5ALFACWTWZE3QCXU3FUY4FKC4P444ORA, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.027920821 (173h38m51.579391283s) [noise=true , address=145.79.227.23:4547 , piece_id=P4WX2ZDHV2Z6BFMF6IACIYW4DXKYFVSULUTF7EK2EMRO5OA5X5RQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.028194272 (173h38m51.579664743s) [noise=true , address=145.79.227.32:4834 , piece_id=CBK2UQ252EPFQDTOYS2X5WAUTJN4L2DTHGIOJXFZXWBOLGPRETXQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.028430262 (173h38m51.579900723s) [noise=true , address=145.79.227.9:4366 , piece_id=YFWF2YL2YPYVMP5DHJKRUGVUUS2ALG4DRJ3HUZCDZM4N3N676FNQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.029687726 (173h38m51.581158177s) [noise=true , address=145.79.227.20:4438 , piece_id=6HY2HDNJAZ552ZKXFJ7VNTQNBAQU5ZQLNKAG2NZZWZQHN6EEDG6Q, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.031784128 (173h38m51.583254589s) [noise=true , address=45.91.116.5:4298 , piece_id=ARGPAZE6CTMM6UXSMPSVHDBEQWKVGKY7WXOVLPB7UH3JZYMDB7JQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.034100830 (173h38m51.585571291s) [noise=true , address=77.90.31.10:4518 , piece_id=MSKQ66KE4WANAE4AWOD2IKY3PQN2YGZ5N4W3NXGKXSOLE7B6H6WQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:54.850200282 (173h38m53.401670733s) [upload=152907]: Next returned nil segment Nov 16 13:42:54.850219118 (173h38m53.401689569s) [upload=152907]: Waiting for error group managing segments... Nov 16 13:42:55.154333032 (173h38m53.705803493s) [noise=true , address=45.91.116.7:4010 , piece_id=EO7IEOCWEJAMZZU4BLIB5ALFACWTWZE3QCXU3FUY4FKC4P444ORA, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:42:55.201689040 (173h38m53.753159501s) [noise=true , address=77.90.31.10:4518 , piece_id=MSKQ66KE4WANAE4AWOD2IKY3PQN2YGZ5N4W3NXGKXSOLE7B6H6WQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:42:55.232313345 (173h38m53.783783816s) [noise=true , address=45.91.116.5:4298 , piece_id=ARGPAZE6CTMM6UXSMPSVHDBEQWKVGKY7WXOVLPB7UH3JZYMDB7JQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:42:55.260281393 (173h38m53.811751854s) [noise=true , address=45.91.116.6:4974 , piece_id=ZTS3H7E6A3VJECLRRKRKSVX5PH3VSH7F2HAUPQSPH56NCI6RTFBQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:00.229127085 (173h38m58.780597546s) [noise=true , address=145.79.227.20:4438 , piece_id=6HY2HDNJAZ552ZKXFJ7VNTQNBAQU5ZQLNKAG2NZZWZQHN6EEDG6Q, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:00.358241119 (173h38m58.909711581s) [noise=true , address=145.79.227.32:4834 , piece_id=CBK2UQ252EPFQDTOYS2X5WAUTJN4L2DTHGIOJXFZXWBOLGPRETXQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:00.646958215 (173h38m59.198428676s) [noise=true , address=145.79.227.31:4798 , piece_id=DPLALMJC4JPC5D7BOK5SOSDFDJFDDPGSJWP4XDDQE22AMFNAKATQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:00.908824080 (173h38m59.460294541s) [noise=true , address=145.79.227.23:4547 , piece_id=P4WX2ZDHV2Z6BFMF6IACIYW4DXKYFVSULUTF7EK2EMRO5OA5X5RQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:01.155346527 (173h38m59.706816989s) [noise=true , address=45.91.116.14:4189 , piece_id=NUMS6Q3F2OUAQ2PSKE2CZK2WKRQLZUZPWVBA3UA6UNKDQACPYSJQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:01.839444794 (173h39m0.390915255s) [noise=true , address=145.79.227.11:4732 , piece_id=GR6SASHNUTHAEOUPK7DWKVPHRGMG72FQMF4RINJ5NTIQQGFCVZZQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:02.591249579 (173h39m1.14272004s) [noise=true , address=145.79.227.10:4402 , piece_id=DNT5QVAMRO5OMBNVTDRRUKACMM7M5472LD5E4ASKYIBKBNWYAVRA, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:02.616126740 (173h39m1.167597202s) [noise=true , address=145.79.227.31:4797 , piece_id=UKP3MHXBSALCWFIHCTSR5JFFXEWWAWGDS4JYUYS2JEREQQKTPJSA, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:02.734605482 (173h39m1.286075943s) [noise=true , address=145.79.227.27:4691 , piece_id=NSX5S2D7ZSA5I57B4EAQEQXEVUMPKNMRKRV3RHWGWJ2DN7SWUPGQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:02.734638004 (173h39m1.286108475s) [seg_pos={0 0}, upload=152907]: Segment reached optimal threshold of 13 pieces. Nov 16 13:43:02.735049748 (173h39m1.286520209s) [noise=true , address=145.79.227.9:4366 , piece_id=YFWF2YL2YPYVMP5DHJKRUGVUUS2ALG4DRJ3HUZCDZM4N3N676FNQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy): protocol: write tcp 188.166.225.170:60100->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:60100->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60100->145.79.227.9:4366: use of closed network connection Nov 16 13:43:02.735060949 (173h39m1.28653141s) [noise=true , address=145.79.227.9:4366 , piece_id=YFWF2YL2YPYVMP5DHJKRUGVUUS2ALG4DRJ3HUZCDZM4N3N676FNQ, seg_pos={0 0}, upload=152907]: Long tail cancellation due to optimal threshold Nov 16 13:43:02.735509193 (173h39m1.286979654s) [noise=true , address=77.90.31.14:4662 , piece_id=S5PPMCKIKSOCLUJ4CEH7ZDW3TVUZ7GTFFEIPCQK6FX25UIOUQHHQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): protocol: write tcp 188.166.225.170:50196->77.90.31.14:4662: use of closed network connection; write tcp 188.166.225.170:50196->77.90.31.14:4662: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50196->77.90.31.14:4662: use of closed network connection Nov 16 13:43:02.735518792 (173h39m1.286989253s) [noise=true , address=77.90.31.14:4662 , piece_id=S5PPMCKIKSOCLUJ4CEH7ZDW3TVUZ7GTFFEIPCQK6FX25UIOUQHHQ, seg_pos={0 0}, upload=152907]: Long tail cancellation due to optimal threshold Nov 16 13:43:02.736224335 (173h39m1.287694796s) [noise=true , address=145.79.227.20:4437 , piece_id=G5P6OYXVAAQDZKTPPHDNZLMYTT6L2TFWKH7N4F7L6742BWHOQGBQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV): protocol: read tcp 188.166.225.170:35300->145.79.227.20:4437: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 13:43:02.736239203 (173h39m1.287709654s) [noise=true , address=145.79.227.20:4437 , piece_id=G5P6OYXVAAQDZKTPPHDNZLMYTT6L2TFWKH7N4F7L6742BWHOQGBQ, seg_pos={0 0}, upload=152907]: Long tail cancellation due to optimal threshold Nov 16 13:43:02.736270833 (173h39m1.287741294s) [seg_pos={0 0}, upload=152907]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 16 13:43:02.736437060 (173h39m1.287907521s) [upload=152907]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 16 13:43:02.736444273 (173h39m1.287914735s) [upload=152907]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 16 13:43:05.031396865 (173h39m3.582867336s) [upload=152907]: Done uploading segments... Nov 16 13:43:05.031422193 (173h39m3.582892654s) [upload=152907]: Upload finished. err: <nil> Duration: 6.0810852s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 11 out of 60 is failed (threshold=6) [...........XX....X........X.........XX....X......X..XX....X.] duration: 4.915768052s trace: https://jaeger.dev.storj.io/trace/69ae5765386935f3 FAIL (2025-11-16 13:49:34 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.915876069s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 14.919930781s trace: https://jaeger.dev.storj.io/trace/129f27707330b16c FAIL (2025-11-16 13:49:32 UTC) orbiter warning: context deadline exceeded Duration: 14.919956221s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 11 out of 60 is failed (threshold=6) [............XX....X........X.........XX....X......X..XX....X] duration: 5.002044655s trace: https://jaeger.dev.storj.io/trace/69ae5765386935f3 FAIL (2025-11-16 13:48:49 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.002068559s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 5.961992384s trace: https://jaeger.dev.storj.io/trace/3c77028f820d12f9 FAIL (2025-11-16 13:48:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, Piece ID: UKP3MHXBSALCWFIHCTSR5JFFXEWWAWGDS4JYUYS2JEREQQKTPJSA) context deadline exceeded; piecestore: (Node ID: 12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, Piece ID: DNT5QVAMRO5OMBNVTDRRUKACMM7M5472LD5E4ASKYIBKBNWYAVRA) context deadline exceeded; piecestore: (Node ID: 17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, Piece ID: DPLALMJC4JPC5D7BOK5SOSDFDJFDDPGSJWP4XDDQE22AMFNAKATQ) context deadline exceeded ======LOGS====== gment. Inline: false Nov 16 13:42:52.666975419 (173h38m51.21844588s) [upload=152907]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 16 13:42:52.666980539 (173h38m51.21845099s) [upload=152907]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 16 13:42:53.024947049 (173h38m51.5764175s) [seg_pos={0 0}, upload=152907]: Begin upload segment... Nov 16 13:42:53.025007074 (173h38m51.576477535s) [seg_pos={0 0}, upload=152907]: Done begin upload segment. Nov 16 13:42:53.025087707 (173h38m51.576558158s) [noise=true , address=77.90.31.14:4662 , piece_id=S5PPMCKIKSOCLUJ4CEH7ZDW3TVUZ7GTFFEIPCQK6FX25UIOUQHHQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.025098377 (173h38m51.576568828s) [noise=true , address=145.79.227.31:4797 , piece_id=UKP3MHXBSALCWFIHCTSR5JFFXEWWAWGDS4JYUYS2JEREQQKTPJSA, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.025961451 (173h38m51.577431912s) [noise=true , address=45.91.116.6:4974 , piece_id=ZTS3H7E6A3VJECLRRKRKSVX5PH3VSH7F2HAUPQSPH56NCI6RTFBQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.026101888 (173h38m51.577572339s) [noise=true , address=145.79.227.10:4402 , piece_id=DNT5QVAMRO5OMBNVTDRRUKACMM7M5472LD5E4ASKYIBKBNWYAVRA, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.026397721 (173h38m51.577868182s) [noise=true , address=145.79.227.11:4732 , piece_id=GR6SASHNUTHAEOUPK7DWKVPHRGMG72FQMF4RINJ5NTIQQGFCVZZQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.026543650 (173h38m51.578014111s) [noise=true , address=145.79.227.20:4437 , piece_id=G5P6OYXVAAQDZKTPPHDNZLMYTT6L2TFWKH7N4F7L6742BWHOQGBQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.026887915 (173h38m51.578358366s) [noise=true , address=45.91.116.14:4189 , piece_id=NUMS6Q3F2OUAQ2PSKE2CZK2WKRQLZUZPWVBA3UA6UNKDQACPYSJQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.027039624 (173h38m51.578510085s) [noise=true , address=145.79.227.27:4691 , piece_id=NSX5S2D7ZSA5I57B4EAQEQXEVUMPKNMRKRV3RHWGWJ2DN7SWUPGQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.027374482 (173h38m51.578844973s) [noise=true , address=145.79.227.31:4798 , piece_id=DPLALMJC4JPC5D7BOK5SOSDFDJFDDPGSJWP4XDDQE22AMFNAKATQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.027472337 (173h38m51.578942798s) [noise=true , address=45.91.116.7:4010 , piece_id=EO7IEOCWEJAMZZU4BLIB5ALFACWTWZE3QCXU3FUY4FKC4P444ORA, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.027920821 (173h38m51.579391283s) [noise=true , address=145.79.227.23:4547 , piece_id=P4WX2ZDHV2Z6BFMF6IACIYW4DXKYFVSULUTF7EK2EMRO5OA5X5RQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.028194272 (173h38m51.579664743s) [noise=true , address=145.79.227.32:4834 , piece_id=CBK2UQ252EPFQDTOYS2X5WAUTJN4L2DTHGIOJXFZXWBOLGPRETXQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.028430262 (173h38m51.579900723s) [noise=true , address=145.79.227.9:4366 , piece_id=YFWF2YL2YPYVMP5DHJKRUGVUUS2ALG4DRJ3HUZCDZM4N3N676FNQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.029687726 (173h38m51.581158177s) [noise=true , address=145.79.227.20:4438 , piece_id=6HY2HDNJAZ552ZKXFJ7VNTQNBAQU5ZQLNKAG2NZZWZQHN6EEDG6Q, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.031784128 (173h38m51.583254589s) [noise=true , address=45.91.116.5:4298 , piece_id=ARGPAZE6CTMM6UXSMPSVHDBEQWKVGKY7WXOVLPB7UH3JZYMDB7JQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.034100830 (173h38m51.585571291s) [noise=true , address=77.90.31.10:4518 , piece_id=MSKQ66KE4WANAE4AWOD2IKY3PQN2YGZ5N4W3NXGKXSOLE7B6H6WQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:54.850200282 (173h38m53.401670733s) [upload=152907]: Next returned nil segment Nov 16 13:42:54.850219118 (173h38m53.401689569s) [upload=152907]: Waiting for error group managing segments... Nov 16 13:42:55.154333032 (173h38m53.705803493s) [noise=true , address=45.91.116.7:4010 , piece_id=EO7IEOCWEJAMZZU4BLIB5ALFACWTWZE3QCXU3FUY4FKC4P444ORA, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:42:55.201689040 (173h38m53.753159501s) [noise=true , address=77.90.31.10:4518 , piece_id=MSKQ66KE4WANAE4AWOD2IKY3PQN2YGZ5N4W3NXGKXSOLE7B6H6WQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:42:55.232313345 (173h38m53.783783816s) [noise=true , address=45.91.116.5:4298 , piece_id=ARGPAZE6CTMM6UXSMPSVHDBEQWKVGKY7WXOVLPB7UH3JZYMDB7JQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:42:55.260281393 (173h38m53.811751854s) [noise=true , address=45.91.116.6:4974 , piece_id=ZTS3H7E6A3VJECLRRKRKSVX5PH3VSH7F2HAUPQSPH56NCI6RTFBQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:00.229127085 (173h38m58.780597546s) [noise=true , address=145.79.227.20:4438 , piece_id=6HY2HDNJAZ552ZKXFJ7VNTQNBAQU5ZQLNKAG2NZZWZQHN6EEDG6Q, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:00.358241119 (173h38m58.909711581s) [noise=true , address=145.79.227.32:4834 , piece_id=CBK2UQ252EPFQDTOYS2X5WAUTJN4L2DTHGIOJXFZXWBOLGPRETXQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:00.646958215 (173h38m59.198428676s) [noise=true , address=145.79.227.31:4798 , piece_id=DPLALMJC4JPC5D7BOK5SOSDFDJFDDPGSJWP4XDDQE22AMFNAKATQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:00.908824080 (173h38m59.460294541s) [noise=true , address=145.79.227.23:4547 , piece_id=P4WX2ZDHV2Z6BFMF6IACIYW4DXKYFVSULUTF7EK2EMRO5OA5X5RQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:01.155346527 (173h38m59.706816989s) [noise=true , address=45.91.116.14:4189 , piece_id=NUMS6Q3F2OUAQ2PSKE2CZK2WKRQLZUZPWVBA3UA6UNKDQACPYSJQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:01.839444794 (173h39m0.390915255s) [noise=true , address=145.79.227.11:4732 , piece_id=GR6SASHNUTHAEOUPK7DWKVPHRGMG72FQMF4RINJ5NTIQQGFCVZZQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:02.591249579 (173h39m1.14272004s) [noise=true , address=145.79.227.10:4402 , piece_id=DNT5QVAMRO5OMBNVTDRRUKACMM7M5472LD5E4ASKYIBKBNWYAVRA, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:02.616126740 (173h39m1.167597202s) [noise=true , address=145.79.227.31:4797 , piece_id=UKP3MHXBSALCWFIHCTSR5JFFXEWWAWGDS4JYUYS2JEREQQKTPJSA, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:02.734605482 (173h39m1.286075943s) [noise=true , address=145.79.227.27:4691 , piece_id=NSX5S2D7ZSA5I57B4EAQEQXEVUMPKNMRKRV3RHWGWJ2DN7SWUPGQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:02.734638004 (173h39m1.286108475s) [seg_pos={0 0}, upload=152907]: Segment reached optimal threshold of 13 pieces. Nov 16 13:43:02.735049748 (173h39m1.286520209s) [noise=true , address=145.79.227.9:4366 , piece_id=YFWF2YL2YPYVMP5DHJKRUGVUUS2ALG4DRJ3HUZCDZM4N3N676FNQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy): protocol: write tcp 188.166.225.170:60100->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:60100->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60100->145.79.227.9:4366: use of closed network connection Nov 16 13:43:02.735060949 (173h39m1.28653141s) [noise=true , address=145.79.227.9:4366 , piece_id=YFWF2YL2YPYVMP5DHJKRUGVUUS2ALG4DRJ3HUZCDZM4N3N676FNQ, seg_pos={0 0}, upload=152907]: Long tail cancellation due to optimal threshold Nov 16 13:43:02.735509193 (173h39m1.286979654s) [noise=true , address=77.90.31.14:4662 , piece_id=S5PPMCKIKSOCLUJ4CEH7ZDW3TVUZ7GTFFEIPCQK6FX25UIOUQHHQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): protocol: write tcp 188.166.225.170:50196->77.90.31.14:4662: use of closed network connection; write tcp 188.166.225.170:50196->77.90.31.14:4662: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50196->77.90.31.14:4662: use of closed network connection Nov 16 13:43:02.735518792 (173h39m1.286989253s) [noise=true , address=77.90.31.14:4662 , piece_id=S5PPMCKIKSOCLUJ4CEH7ZDW3TVUZ7GTFFEIPCQK6FX25UIOUQHHQ, seg_pos={0 0}, upload=152907]: Long tail cancellation due to optimal threshold Nov 16 13:43:02.736224335 (173h39m1.287694796s) [noise=true , address=145.79.227.20:4437 , piece_id=G5P6OYXVAAQDZKTPPHDNZLMYTT6L2TFWKH7N4F7L6742BWHOQGBQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV): protocol: read tcp 188.166.225.170:35300->145.79.227.20:4437: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 13:43:02.736239203 (173h39m1.287709654s) [noise=true , address=145.79.227.20:4437 , piece_id=G5P6OYXVAAQDZKTPPHDNZLMYTT6L2TFWKH7N4F7L6742BWHOQGBQ, seg_pos={0 0}, upload=152907]: Long tail cancellation due to optimal threshold Nov 16 13:43:02.736270833 (173h39m1.287741294s) [seg_pos={0 0}, upload=152907]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 16 13:43:02.736437060 (173h39m1.287907521s) [upload=152907]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 16 13:43:02.736444273 (173h39m1.287914735s) [upload=152907]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 16 13:43:05.031396865 (173h39m3.582867336s) [upload=152907]: Done uploading segments... Nov 16 13:43:05.031422193 (173h39m3.582892654s) [upload=152907]: Upload finished. err: <nil> Duration: 5.962027832s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 15.408873752s trace: https://jaeger.dev.storj.io/trace/129f27707330b16c FAIL (2025-11-16 13:48:20 UTC) orbiter warning: context deadline exceeded Duration: 15.408899069s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 10 out of 60 is failed (threshold=6) [.............XX....X........X.........XX....X......X..XX....] duration: 3.953656639s trace: https://jaeger.dev.storj.io/trace/e4f9648cfff91e2 FAIL (2025-11-16 13:48:03 UTC) orbiter warning: context deadline exceeded Duration: 3.953687747s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 14.312914206s trace: https://jaeger.dev.storj.io/trace/129f27707330b16c FAIL (2025-11-16 13:47:32 UTC) orbiter warning: context deadline exceeded Duration: 14.312943333s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 6.142926588s trace: https://jaeger.dev.storj.io/trace/3c77028f820d12f9 FAIL (2025-11-16 13:47:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, Piece ID: UKP3MHXBSALCWFIHCTSR5JFFXEWWAWGDS4JYUYS2JEREQQKTPJSA) context deadline exceeded; piecestore: (Node ID: 12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, Piece ID: DNT5QVAMRO5OMBNVTDRRUKACMM7M5472LD5E4ASKYIBKBNWYAVRA) context deadline exceeded; piecestore: (Node ID: 17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, Piece ID: DPLALMJC4JPC5D7BOK5SOSDFDJFDDPGSJWP4XDDQE22AMFNAKATQ) context deadline exceeded ======LOGS====== gment. Inline: false Nov 16 13:42:52.666975419 (173h38m51.21844588s) [upload=152907]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 16 13:42:52.666980539 (173h38m51.21845099s) [upload=152907]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 16 13:42:53.024947049 (173h38m51.5764175s) [seg_pos={0 0}, upload=152907]: Begin upload segment... Nov 16 13:42:53.025007074 (173h38m51.576477535s) [seg_pos={0 0}, upload=152907]: Done begin upload segment. Nov 16 13:42:53.025087707 (173h38m51.576558158s) [noise=true , address=77.90.31.14:4662 , piece_id=S5PPMCKIKSOCLUJ4CEH7ZDW3TVUZ7GTFFEIPCQK6FX25UIOUQHHQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.025098377 (173h38m51.576568828s) [noise=true , address=145.79.227.31:4797 , piece_id=UKP3MHXBSALCWFIHCTSR5JFFXEWWAWGDS4JYUYS2JEREQQKTPJSA, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.025961451 (173h38m51.577431912s) [noise=true , address=45.91.116.6:4974 , piece_id=ZTS3H7E6A3VJECLRRKRKSVX5PH3VSH7F2HAUPQSPH56NCI6RTFBQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.026101888 (173h38m51.577572339s) [noise=true , address=145.79.227.10:4402 , piece_id=DNT5QVAMRO5OMBNVTDRRUKACMM7M5472LD5E4ASKYIBKBNWYAVRA, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.026397721 (173h38m51.577868182s) [noise=true , address=145.79.227.11:4732 , piece_id=GR6SASHNUTHAEOUPK7DWKVPHRGMG72FQMF4RINJ5NTIQQGFCVZZQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.026543650 (173h38m51.578014111s) [noise=true , address=145.79.227.20:4437 , piece_id=G5P6OYXVAAQDZKTPPHDNZLMYTT6L2TFWKH7N4F7L6742BWHOQGBQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.026887915 (173h38m51.578358366s) [noise=true , address=45.91.116.14:4189 , piece_id=NUMS6Q3F2OUAQ2PSKE2CZK2WKRQLZUZPWVBA3UA6UNKDQACPYSJQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.027039624 (173h38m51.578510085s) [noise=true , address=145.79.227.27:4691 , piece_id=NSX5S2D7ZSA5I57B4EAQEQXEVUMPKNMRKRV3RHWGWJ2DN7SWUPGQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.027374482 (173h38m51.578844973s) [noise=true , address=145.79.227.31:4798 , piece_id=DPLALMJC4JPC5D7BOK5SOSDFDJFDDPGSJWP4XDDQE22AMFNAKATQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.027472337 (173h38m51.578942798s) [noise=true , address=45.91.116.7:4010 , piece_id=EO7IEOCWEJAMZZU4BLIB5ALFACWTWZE3QCXU3FUY4FKC4P444ORA, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.027920821 (173h38m51.579391283s) [noise=true , address=145.79.227.23:4547 , piece_id=P4WX2ZDHV2Z6BFMF6IACIYW4DXKYFVSULUTF7EK2EMRO5OA5X5RQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.028194272 (173h38m51.579664743s) [noise=true , address=145.79.227.32:4834 , piece_id=CBK2UQ252EPFQDTOYS2X5WAUTJN4L2DTHGIOJXFZXWBOLGPRETXQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.028430262 (173h38m51.579900723s) [noise=true , address=145.79.227.9:4366 , piece_id=YFWF2YL2YPYVMP5DHJKRUGVUUS2ALG4DRJ3HUZCDZM4N3N676FNQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.029687726 (173h38m51.581158177s) [noise=true , address=145.79.227.20:4438 , piece_id=6HY2HDNJAZ552ZKXFJ7VNTQNBAQU5ZQLNKAG2NZZWZQHN6EEDG6Q, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.031784128 (173h38m51.583254589s) [noise=true , address=45.91.116.5:4298 , piece_id=ARGPAZE6CTMM6UXSMPSVHDBEQWKVGKY7WXOVLPB7UH3JZYMDB7JQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.034100830 (173h38m51.585571291s) [noise=true , address=77.90.31.10:4518 , piece_id=MSKQ66KE4WANAE4AWOD2IKY3PQN2YGZ5N4W3NXGKXSOLE7B6H6WQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:54.850200282 (173h38m53.401670733s) [upload=152907]: Next returned nil segment Nov 16 13:42:54.850219118 (173h38m53.401689569s) [upload=152907]: Waiting for error group managing segments... Nov 16 13:42:55.154333032 (173h38m53.705803493s) [noise=true , address=45.91.116.7:4010 , piece_id=EO7IEOCWEJAMZZU4BLIB5ALFACWTWZE3QCXU3FUY4FKC4P444ORA, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:42:55.201689040 (173h38m53.753159501s) [noise=true , address=77.90.31.10:4518 , piece_id=MSKQ66KE4WANAE4AWOD2IKY3PQN2YGZ5N4W3NXGKXSOLE7B6H6WQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:42:55.232313345 (173h38m53.783783816s) [noise=true , address=45.91.116.5:4298 , piece_id=ARGPAZE6CTMM6UXSMPSVHDBEQWKVGKY7WXOVLPB7UH3JZYMDB7JQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:42:55.260281393 (173h38m53.811751854s) [noise=true , address=45.91.116.6:4974 , piece_id=ZTS3H7E6A3VJECLRRKRKSVX5PH3VSH7F2HAUPQSPH56NCI6RTFBQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:00.229127085 (173h38m58.780597546s) [noise=true , address=145.79.227.20:4438 , piece_id=6HY2HDNJAZ552ZKXFJ7VNTQNBAQU5ZQLNKAG2NZZWZQHN6EEDG6Q, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:00.358241119 (173h38m58.909711581s) [noise=true , address=145.79.227.32:4834 , piece_id=CBK2UQ252EPFQDTOYS2X5WAUTJN4L2DTHGIOJXFZXWBOLGPRETXQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:00.646958215 (173h38m59.198428676s) [noise=true , address=145.79.227.31:4798 , piece_id=DPLALMJC4JPC5D7BOK5SOSDFDJFDDPGSJWP4XDDQE22AMFNAKATQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:00.908824080 (173h38m59.460294541s) [noise=true , address=145.79.227.23:4547 , piece_id=P4WX2ZDHV2Z6BFMF6IACIYW4DXKYFVSULUTF7EK2EMRO5OA5X5RQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:01.155346527 (173h38m59.706816989s) [noise=true , address=45.91.116.14:4189 , piece_id=NUMS6Q3F2OUAQ2PSKE2CZK2WKRQLZUZPWVBA3UA6UNKDQACPYSJQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:01.839444794 (173h39m0.390915255s) [noise=true , address=145.79.227.11:4732 , piece_id=GR6SASHNUTHAEOUPK7DWKVPHRGMG72FQMF4RINJ5NTIQQGFCVZZQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:02.591249579 (173h39m1.14272004s) [noise=true , address=145.79.227.10:4402 , piece_id=DNT5QVAMRO5OMBNVTDRRUKACMM7M5472LD5E4ASKYIBKBNWYAVRA, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:02.616126740 (173h39m1.167597202s) [noise=true , address=145.79.227.31:4797 , piece_id=UKP3MHXBSALCWFIHCTSR5JFFXEWWAWGDS4JYUYS2JEREQQKTPJSA, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:02.734605482 (173h39m1.286075943s) [noise=true , address=145.79.227.27:4691 , piece_id=NSX5S2D7ZSA5I57B4EAQEQXEVUMPKNMRKRV3RHWGWJ2DN7SWUPGQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:02.734638004 (173h39m1.286108475s) [seg_pos={0 0}, upload=152907]: Segment reached optimal threshold of 13 pieces. Nov 16 13:43:02.735049748 (173h39m1.286520209s) [noise=true , address=145.79.227.9:4366 , piece_id=YFWF2YL2YPYVMP5DHJKRUGVUUS2ALG4DRJ3HUZCDZM4N3N676FNQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy): protocol: write tcp 188.166.225.170:60100->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:60100->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60100->145.79.227.9:4366: use of closed network connection Nov 16 13:43:02.735060949 (173h39m1.28653141s) [noise=true , address=145.79.227.9:4366 , piece_id=YFWF2YL2YPYVMP5DHJKRUGVUUS2ALG4DRJ3HUZCDZM4N3N676FNQ, seg_pos={0 0}, upload=152907]: Long tail cancellation due to optimal threshold Nov 16 13:43:02.735509193 (173h39m1.286979654s) [noise=true , address=77.90.31.14:4662 , piece_id=S5PPMCKIKSOCLUJ4CEH7ZDW3TVUZ7GTFFEIPCQK6FX25UIOUQHHQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): protocol: write tcp 188.166.225.170:50196->77.90.31.14:4662: use of closed network connection; write tcp 188.166.225.170:50196->77.90.31.14:4662: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50196->77.90.31.14:4662: use of closed network connection Nov 16 13:43:02.735518792 (173h39m1.286989253s) [noise=true , address=77.90.31.14:4662 , piece_id=S5PPMCKIKSOCLUJ4CEH7ZDW3TVUZ7GTFFEIPCQK6FX25UIOUQHHQ, seg_pos={0 0}, upload=152907]: Long tail cancellation due to optimal threshold Nov 16 13:43:02.736224335 (173h39m1.287694796s) [noise=true , address=145.79.227.20:4437 , piece_id=G5P6OYXVAAQDZKTPPHDNZLMYTT6L2TFWKH7N4F7L6742BWHOQGBQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV): protocol: read tcp 188.166.225.170:35300->145.79.227.20:4437: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 13:43:02.736239203 (173h39m1.287709654s) [noise=true , address=145.79.227.20:4437 , piece_id=G5P6OYXVAAQDZKTPPHDNZLMYTT6L2TFWKH7N4F7L6742BWHOQGBQ, seg_pos={0 0}, upload=152907]: Long tail cancellation due to optimal threshold Nov 16 13:43:02.736270833 (173h39m1.287741294s) [seg_pos={0 0}, upload=152907]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 16 13:43:02.736437060 (173h39m1.287907521s) [upload=152907]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 16 13:43:02.736444273 (173h39m1.287914735s) [upload=152907]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 16 13:43:05.031396865 (173h39m3.582867336s) [upload=152907]: Done uploading segments... Nov 16 13:43:05.031422193 (173h39m3.582892654s) [upload=152907]: Upload finished. err: <nil> Duration: 6.142967385s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 10 out of 60 is failed (threshold=6) [..............XX....X........X.........XX....X......X..XX...] duration: 5.001712599s trace: https://jaeger.dev.storj.io/trace/e4f9648cfff91e2 FAIL (2025-11-16 13:46:58 UTC) orbiter warning: context deadline exceeded Duration: 5.001782764s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 6.027815594s trace: https://jaeger.dev.storj.io/trace/3c77028f820d12f9 FAIL (2025-11-16 13:46:45 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, Piece ID: UKP3MHXBSALCWFIHCTSR5JFFXEWWAWGDS4JYUYS2JEREQQKTPJSA) context deadline exceeded; piecestore: (Node ID: 12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, Piece ID: DNT5QVAMRO5OMBNVTDRRUKACMM7M5472LD5E4ASKYIBKBNWYAVRA) context deadline exceeded; piecestore: (Node ID: 17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, Piece ID: DPLALMJC4JPC5D7BOK5SOSDFDJFDDPGSJWP4XDDQE22AMFNAKATQ) context deadline exceeded ======LOGS====== gment. Inline: false Nov 16 13:42:52.666975419 (173h38m51.21844588s) [upload=152907]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 16 13:42:52.666980539 (173h38m51.21845099s) [upload=152907]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 16 13:42:53.024947049 (173h38m51.5764175s) [seg_pos={0 0}, upload=152907]: Begin upload segment... Nov 16 13:42:53.025007074 (173h38m51.576477535s) [seg_pos={0 0}, upload=152907]: Done begin upload segment. Nov 16 13:42:53.025087707 (173h38m51.576558158s) [noise=true , address=77.90.31.14:4662 , piece_id=S5PPMCKIKSOCLUJ4CEH7ZDW3TVUZ7GTFFEIPCQK6FX25UIOUQHHQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.025098377 (173h38m51.576568828s) [noise=true , address=145.79.227.31:4797 , piece_id=UKP3MHXBSALCWFIHCTSR5JFFXEWWAWGDS4JYUYS2JEREQQKTPJSA, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.025961451 (173h38m51.577431912s) [noise=true , address=45.91.116.6:4974 , piece_id=ZTS3H7E6A3VJECLRRKRKSVX5PH3VSH7F2HAUPQSPH56NCI6RTFBQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.026101888 (173h38m51.577572339s) [noise=true , address=145.79.227.10:4402 , piece_id=DNT5QVAMRO5OMBNVTDRRUKACMM7M5472LD5E4ASKYIBKBNWYAVRA, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.026397721 (173h38m51.577868182s) [noise=true , address=145.79.227.11:4732 , piece_id=GR6SASHNUTHAEOUPK7DWKVPHRGMG72FQMF4RINJ5NTIQQGFCVZZQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.026543650 (173h38m51.578014111s) [noise=true , address=145.79.227.20:4437 , piece_id=G5P6OYXVAAQDZKTPPHDNZLMYTT6L2TFWKH7N4F7L6742BWHOQGBQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.026887915 (173h38m51.578358366s) [noise=true , address=45.91.116.14:4189 , piece_id=NUMS6Q3F2OUAQ2PSKE2CZK2WKRQLZUZPWVBA3UA6UNKDQACPYSJQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.027039624 (173h38m51.578510085s) [noise=true , address=145.79.227.27:4691 , piece_id=NSX5S2D7ZSA5I57B4EAQEQXEVUMPKNMRKRV3RHWGWJ2DN7SWUPGQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.027374482 (173h38m51.578844973s) [noise=true , address=145.79.227.31:4798 , piece_id=DPLALMJC4JPC5D7BOK5SOSDFDJFDDPGSJWP4XDDQE22AMFNAKATQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.027472337 (173h38m51.578942798s) [noise=true , address=45.91.116.7:4010 , piece_id=EO7IEOCWEJAMZZU4BLIB5ALFACWTWZE3QCXU3FUY4FKC4P444ORA, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.027920821 (173h38m51.579391283s) [noise=true , address=145.79.227.23:4547 , piece_id=P4WX2ZDHV2Z6BFMF6IACIYW4DXKYFVSULUTF7EK2EMRO5OA5X5RQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.028194272 (173h38m51.579664743s) [noise=true , address=145.79.227.32:4834 , piece_id=CBK2UQ252EPFQDTOYS2X5WAUTJN4L2DTHGIOJXFZXWBOLGPRETXQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.028430262 (173h38m51.579900723s) [noise=true , address=145.79.227.9:4366 , piece_id=YFWF2YL2YPYVMP5DHJKRUGVUUS2ALG4DRJ3HUZCDZM4N3N676FNQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.029687726 (173h38m51.581158177s) [noise=true , address=145.79.227.20:4438 , piece_id=6HY2HDNJAZ552ZKXFJ7VNTQNBAQU5ZQLNKAG2NZZWZQHN6EEDG6Q, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.031784128 (173h38m51.583254589s) [noise=true , address=45.91.116.5:4298 , piece_id=ARGPAZE6CTMM6UXSMPSVHDBEQWKVGKY7WXOVLPB7UH3JZYMDB7JQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.034100830 (173h38m51.585571291s) [noise=true , address=77.90.31.10:4518 , piece_id=MSKQ66KE4WANAE4AWOD2IKY3PQN2YGZ5N4W3NXGKXSOLE7B6H6WQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:54.850200282 (173h38m53.401670733s) [upload=152907]: Next returned nil segment Nov 16 13:42:54.850219118 (173h38m53.401689569s) [upload=152907]: Waiting for error group managing segments... Nov 16 13:42:55.154333032 (173h38m53.705803493s) [noise=true , address=45.91.116.7:4010 , piece_id=EO7IEOCWEJAMZZU4BLIB5ALFACWTWZE3QCXU3FUY4FKC4P444ORA, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:42:55.201689040 (173h38m53.753159501s) [noise=true , address=77.90.31.10:4518 , piece_id=MSKQ66KE4WANAE4AWOD2IKY3PQN2YGZ5N4W3NXGKXSOLE7B6H6WQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:42:55.232313345 (173h38m53.783783816s) [noise=true , address=45.91.116.5:4298 , piece_id=ARGPAZE6CTMM6UXSMPSVHDBEQWKVGKY7WXOVLPB7UH3JZYMDB7JQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:42:55.260281393 (173h38m53.811751854s) [noise=true , address=45.91.116.6:4974 , piece_id=ZTS3H7E6A3VJECLRRKRKSVX5PH3VSH7F2HAUPQSPH56NCI6RTFBQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:00.229127085 (173h38m58.780597546s) [noise=true , address=145.79.227.20:4438 , piece_id=6HY2HDNJAZ552ZKXFJ7VNTQNBAQU5ZQLNKAG2NZZWZQHN6EEDG6Q, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:00.358241119 (173h38m58.909711581s) [noise=true , address=145.79.227.32:4834 , piece_id=CBK2UQ252EPFQDTOYS2X5WAUTJN4L2DTHGIOJXFZXWBOLGPRETXQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:00.646958215 (173h38m59.198428676s) [noise=true , address=145.79.227.31:4798 , piece_id=DPLALMJC4JPC5D7BOK5SOSDFDJFDDPGSJWP4XDDQE22AMFNAKATQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:00.908824080 (173h38m59.460294541s) [noise=true , address=145.79.227.23:4547 , piece_id=P4WX2ZDHV2Z6BFMF6IACIYW4DXKYFVSULUTF7EK2EMRO5OA5X5RQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:01.155346527 (173h38m59.706816989s) [noise=true , address=45.91.116.14:4189 , piece_id=NUMS6Q3F2OUAQ2PSKE2CZK2WKRQLZUZPWVBA3UA6UNKDQACPYSJQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:01.839444794 (173h39m0.390915255s) [noise=true , address=145.79.227.11:4732 , piece_id=GR6SASHNUTHAEOUPK7DWKVPHRGMG72FQMF4RINJ5NTIQQGFCVZZQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:02.591249579 (173h39m1.14272004s) [noise=true , address=145.79.227.10:4402 , piece_id=DNT5QVAMRO5OMBNVTDRRUKACMM7M5472LD5E4ASKYIBKBNWYAVRA, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:02.616126740 (173h39m1.167597202s) [noise=true , address=145.79.227.31:4797 , piece_id=UKP3MHXBSALCWFIHCTSR5JFFXEWWAWGDS4JYUYS2JEREQQKTPJSA, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:02.734605482 (173h39m1.286075943s) [noise=true , address=145.79.227.27:4691 , piece_id=NSX5S2D7ZSA5I57B4EAQEQXEVUMPKNMRKRV3RHWGWJ2DN7SWUPGQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:02.734638004 (173h39m1.286108475s) [seg_pos={0 0}, upload=152907]: Segment reached optimal threshold of 13 pieces. Nov 16 13:43:02.735049748 (173h39m1.286520209s) [noise=true , address=145.79.227.9:4366 , piece_id=YFWF2YL2YPYVMP5DHJKRUGVUUS2ALG4DRJ3HUZCDZM4N3N676FNQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy): protocol: write tcp 188.166.225.170:60100->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:60100->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60100->145.79.227.9:4366: use of closed network connection Nov 16 13:43:02.735060949 (173h39m1.28653141s) [noise=true , address=145.79.227.9:4366 , piece_id=YFWF2YL2YPYVMP5DHJKRUGVUUS2ALG4DRJ3HUZCDZM4N3N676FNQ, seg_pos={0 0}, upload=152907]: Long tail cancellation due to optimal threshold Nov 16 13:43:02.735509193 (173h39m1.286979654s) [noise=true , address=77.90.31.14:4662 , piece_id=S5PPMCKIKSOCLUJ4CEH7ZDW3TVUZ7GTFFEIPCQK6FX25UIOUQHHQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): protocol: write tcp 188.166.225.170:50196->77.90.31.14:4662: use of closed network connection; write tcp 188.166.225.170:50196->77.90.31.14:4662: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50196->77.90.31.14:4662: use of closed network connection Nov 16 13:43:02.735518792 (173h39m1.286989253s) [noise=true , address=77.90.31.14:4662 , piece_id=S5PPMCKIKSOCLUJ4CEH7ZDW3TVUZ7GTFFEIPCQK6FX25UIOUQHHQ, seg_pos={0 0}, upload=152907]: Long tail cancellation due to optimal threshold Nov 16 13:43:02.736224335 (173h39m1.287694796s) [noise=true , address=145.79.227.20:4437 , piece_id=G5P6OYXVAAQDZKTPPHDNZLMYTT6L2TFWKH7N4F7L6742BWHOQGBQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV): protocol: read tcp 188.166.225.170:35300->145.79.227.20:4437: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 13:43:02.736239203 (173h39m1.287709654s) [noise=true , address=145.79.227.20:4437 , piece_id=G5P6OYXVAAQDZKTPPHDNZLMYTT6L2TFWKH7N4F7L6742BWHOQGBQ, seg_pos={0 0}, upload=152907]: Long tail cancellation due to optimal threshold Nov 16 13:43:02.736270833 (173h39m1.287741294s) [seg_pos={0 0}, upload=152907]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 16 13:43:02.736437060 (173h39m1.287907521s) [upload=152907]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 16 13:43:02.736444273 (173h39m1.287914735s) [upload=152907]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 16 13:43:05.031396865 (173h39m3.582867336s) [upload=152907]: Done uploading segments... Nov 16 13:43:05.031422193 (173h39m3.582892654s) [upload=152907]: Upload finished. err: <nil> Duration: 6.027845331s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 6.182433884s trace: https://jaeger.dev.storj.io/trace/129f27707330b16c FAIL (2025-11-16 13:46:45 UTC) orbiter warning: context deadline exceeded Duration: 6.182462569s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 11.914111027s trace: https://jaeger.dev.storj.io/trace/129f27707330b16c FAIL (2025-11-16 13:46:25 UTC) orbiter warning: context deadline exceeded Duration: 11.914130634s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 6.447705942s trace: https://jaeger.dev.storj.io/trace/3c77028f820d12f9 FAIL (2025-11-16 13:46:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, Piece ID: UKP3MHXBSALCWFIHCTSR5JFFXEWWAWGDS4JYUYS2JEREQQKTPJSA) context deadline exceeded; piecestore: (Node ID: 12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, Piece ID: DNT5QVAMRO5OMBNVTDRRUKACMM7M5472LD5E4ASKYIBKBNWYAVRA) context deadline exceeded; piecestore: (Node ID: 17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, Piece ID: DPLALMJC4JPC5D7BOK5SOSDFDJFDDPGSJWP4XDDQE22AMFNAKATQ) context deadline exceeded ======LOGS====== gment. Inline: false Nov 16 13:42:52.666975419 (173h38m51.21844588s) [upload=152907]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 16 13:42:52.666980539 (173h38m51.21845099s) [upload=152907]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 16 13:42:53.024947049 (173h38m51.5764175s) [seg_pos={0 0}, upload=152907]: Begin upload segment... Nov 16 13:42:53.025007074 (173h38m51.576477535s) [seg_pos={0 0}, upload=152907]: Done begin upload segment. Nov 16 13:42:53.025087707 (173h38m51.576558158s) [noise=true , address=77.90.31.14:4662 , piece_id=S5PPMCKIKSOCLUJ4CEH7ZDW3TVUZ7GTFFEIPCQK6FX25UIOUQHHQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.025098377 (173h38m51.576568828s) [noise=true , address=145.79.227.31:4797 , piece_id=UKP3MHXBSALCWFIHCTSR5JFFXEWWAWGDS4JYUYS2JEREQQKTPJSA, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.025961451 (173h38m51.577431912s) [noise=true , address=45.91.116.6:4974 , piece_id=ZTS3H7E6A3VJECLRRKRKSVX5PH3VSH7F2HAUPQSPH56NCI6RTFBQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.026101888 (173h38m51.577572339s) [noise=true , address=145.79.227.10:4402 , piece_id=DNT5QVAMRO5OMBNVTDRRUKACMM7M5472LD5E4ASKYIBKBNWYAVRA, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.026397721 (173h38m51.577868182s) [noise=true , address=145.79.227.11:4732 , piece_id=GR6SASHNUTHAEOUPK7DWKVPHRGMG72FQMF4RINJ5NTIQQGFCVZZQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.026543650 (173h38m51.578014111s) [noise=true , address=145.79.227.20:4437 , piece_id=G5P6OYXVAAQDZKTPPHDNZLMYTT6L2TFWKH7N4F7L6742BWHOQGBQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.026887915 (173h38m51.578358366s) [noise=true , address=45.91.116.14:4189 , piece_id=NUMS6Q3F2OUAQ2PSKE2CZK2WKRQLZUZPWVBA3UA6UNKDQACPYSJQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.027039624 (173h38m51.578510085s) [noise=true , address=145.79.227.27:4691 , piece_id=NSX5S2D7ZSA5I57B4EAQEQXEVUMPKNMRKRV3RHWGWJ2DN7SWUPGQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.027374482 (173h38m51.578844973s) [noise=true , address=145.79.227.31:4798 , piece_id=DPLALMJC4JPC5D7BOK5SOSDFDJFDDPGSJWP4XDDQE22AMFNAKATQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.027472337 (173h38m51.578942798s) [noise=true , address=45.91.116.7:4010 , piece_id=EO7IEOCWEJAMZZU4BLIB5ALFACWTWZE3QCXU3FUY4FKC4P444ORA, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.027920821 (173h38m51.579391283s) [noise=true , address=145.79.227.23:4547 , piece_id=P4WX2ZDHV2Z6BFMF6IACIYW4DXKYFVSULUTF7EK2EMRO5OA5X5RQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.028194272 (173h38m51.579664743s) [noise=true , address=145.79.227.32:4834 , piece_id=CBK2UQ252EPFQDTOYS2X5WAUTJN4L2DTHGIOJXFZXWBOLGPRETXQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.028430262 (173h38m51.579900723s) [noise=true , address=145.79.227.9:4366 , piece_id=YFWF2YL2YPYVMP5DHJKRUGVUUS2ALG4DRJ3HUZCDZM4N3N676FNQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.029687726 (173h38m51.581158177s) [noise=true , address=145.79.227.20:4438 , piece_id=6HY2HDNJAZ552ZKXFJ7VNTQNBAQU5ZQLNKAG2NZZWZQHN6EEDG6Q, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.031784128 (173h38m51.583254589s) [noise=true , address=45.91.116.5:4298 , piece_id=ARGPAZE6CTMM6UXSMPSVHDBEQWKVGKY7WXOVLPB7UH3JZYMDB7JQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.034100830 (173h38m51.585571291s) [noise=true , address=77.90.31.10:4518 , piece_id=MSKQ66KE4WANAE4AWOD2IKY3PQN2YGZ5N4W3NXGKXSOLE7B6H6WQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:54.850200282 (173h38m53.401670733s) [upload=152907]: Next returned nil segment Nov 16 13:42:54.850219118 (173h38m53.401689569s) [upload=152907]: Waiting for error group managing segments... Nov 16 13:42:55.154333032 (173h38m53.705803493s) [noise=true , address=45.91.116.7:4010 , piece_id=EO7IEOCWEJAMZZU4BLIB5ALFACWTWZE3QCXU3FUY4FKC4P444ORA, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:42:55.201689040 (173h38m53.753159501s) [noise=true , address=77.90.31.10:4518 , piece_id=MSKQ66KE4WANAE4AWOD2IKY3PQN2YGZ5N4W3NXGKXSOLE7B6H6WQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:42:55.232313345 (173h38m53.783783816s) [noise=true , address=45.91.116.5:4298 , piece_id=ARGPAZE6CTMM6UXSMPSVHDBEQWKVGKY7WXOVLPB7UH3JZYMDB7JQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:42:55.260281393 (173h38m53.811751854s) [noise=true , address=45.91.116.6:4974 , piece_id=ZTS3H7E6A3VJECLRRKRKSVX5PH3VSH7F2HAUPQSPH56NCI6RTFBQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:00.229127085 (173h38m58.780597546s) [noise=true , address=145.79.227.20:4438 , piece_id=6HY2HDNJAZ552ZKXFJ7VNTQNBAQU5ZQLNKAG2NZZWZQHN6EEDG6Q, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:00.358241119 (173h38m58.909711581s) [noise=true , address=145.79.227.32:4834 , piece_id=CBK2UQ252EPFQDTOYS2X5WAUTJN4L2DTHGIOJXFZXWBOLGPRETXQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:00.646958215 (173h38m59.198428676s) [noise=true , address=145.79.227.31:4798 , piece_id=DPLALMJC4JPC5D7BOK5SOSDFDJFDDPGSJWP4XDDQE22AMFNAKATQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:00.908824080 (173h38m59.460294541s) [noise=true , address=145.79.227.23:4547 , piece_id=P4WX2ZDHV2Z6BFMF6IACIYW4DXKYFVSULUTF7EK2EMRO5OA5X5RQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:01.155346527 (173h38m59.706816989s) [noise=true , address=45.91.116.14:4189 , piece_id=NUMS6Q3F2OUAQ2PSKE2CZK2WKRQLZUZPWVBA3UA6UNKDQACPYSJQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:01.839444794 (173h39m0.390915255s) [noise=true , address=145.79.227.11:4732 , piece_id=GR6SASHNUTHAEOUPK7DWKVPHRGMG72FQMF4RINJ5NTIQQGFCVZZQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:02.591249579 (173h39m1.14272004s) [noise=true , address=145.79.227.10:4402 , piece_id=DNT5QVAMRO5OMBNVTDRRUKACMM7M5472LD5E4ASKYIBKBNWYAVRA, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:02.616126740 (173h39m1.167597202s) [noise=true , address=145.79.227.31:4797 , piece_id=UKP3MHXBSALCWFIHCTSR5JFFXEWWAWGDS4JYUYS2JEREQQKTPJSA, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:02.734605482 (173h39m1.286075943s) [noise=true , address=145.79.227.27:4691 , piece_id=NSX5S2D7ZSA5I57B4EAQEQXEVUMPKNMRKRV3RHWGWJ2DN7SWUPGQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:02.734638004 (173h39m1.286108475s) [seg_pos={0 0}, upload=152907]: Segment reached optimal threshold of 13 pieces. Nov 16 13:43:02.735049748 (173h39m1.286520209s) [noise=true , address=145.79.227.9:4366 , piece_id=YFWF2YL2YPYVMP5DHJKRUGVUUS2ALG4DRJ3HUZCDZM4N3N676FNQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy): protocol: write tcp 188.166.225.170:60100->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:60100->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60100->145.79.227.9:4366: use of closed network connection Nov 16 13:43:02.735060949 (173h39m1.28653141s) [noise=true , address=145.79.227.9:4366 , piece_id=YFWF2YL2YPYVMP5DHJKRUGVUUS2ALG4DRJ3HUZCDZM4N3N676FNQ, seg_pos={0 0}, upload=152907]: Long tail cancellation due to optimal threshold Nov 16 13:43:02.735509193 (173h39m1.286979654s) [noise=true , address=77.90.31.14:4662 , piece_id=S5PPMCKIKSOCLUJ4CEH7ZDW3TVUZ7GTFFEIPCQK6FX25UIOUQHHQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): protocol: write tcp 188.166.225.170:50196->77.90.31.14:4662: use of closed network connection; write tcp 188.166.225.170:50196->77.90.31.14:4662: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50196->77.90.31.14:4662: use of closed network connection Nov 16 13:43:02.735518792 (173h39m1.286989253s) [noise=true , address=77.90.31.14:4662 , piece_id=S5PPMCKIKSOCLUJ4CEH7ZDW3TVUZ7GTFFEIPCQK6FX25UIOUQHHQ, seg_pos={0 0}, upload=152907]: Long tail cancellation due to optimal threshold Nov 16 13:43:02.736224335 (173h39m1.287694796s) [noise=true , address=145.79.227.20:4437 , piece_id=G5P6OYXVAAQDZKTPPHDNZLMYTT6L2TFWKH7N4F7L6742BWHOQGBQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV): protocol: read tcp 188.166.225.170:35300->145.79.227.20:4437: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 13:43:02.736239203 (173h39m1.287709654s) [noise=true , address=145.79.227.20:4437 , piece_id=G5P6OYXVAAQDZKTPPHDNZLMYTT6L2TFWKH7N4F7L6742BWHOQGBQ, seg_pos={0 0}, upload=152907]: Long tail cancellation due to optimal threshold Nov 16 13:43:02.736270833 (173h39m1.287741294s) [seg_pos={0 0}, upload=152907]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 16 13:43:02.736437060 (173h39m1.287907521s) [upload=152907]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 16 13:43:02.736444273 (173h39m1.287914735s) [upload=152907]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 16 13:43:05.031396865 (173h39m3.582867336s) [upload=152907]: Done uploading segments... Nov 16 13:43:05.031422193 (173h39m3.582892654s) [upload=152907]: Upload finished. err: <nil> Duration: 6.447735157s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 14.44452948s trace: https://jaeger.dev.storj.io/trace/129f27707330b16c FAIL (2025-11-16 13:45:57 UTC) orbiter warning: context deadline exceeded Duration: 14.44456577s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 10 out of 60 is failed (threshold=6) [...............XX....X........X.........XX....X......X..XX..] duration: 3.421572268s trace: https://jaeger.dev.storj.io/trace/e4f9648cfff91e2 FAIL (2025-11-16 13:45:49 UTC) orbiter warning: context deadline exceeded Duration: 3.421615651s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 7.078124462s trace: https://jaeger.dev.storj.io/trace/129f27707330b16c FAIL (2025-11-16 13:45:30 UTC) orbiter warning: context deadline exceeded Duration: 7.078156482s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 10 out of 60 is failed (threshold=6) [................XX....X........X.........XX....X......X..XX.] duration: 3.466487712s trace: https://jaeger.dev.storj.io/trace/e4f9648cfff91e2 FAIL (2025-11-16 13:45:25 UTC) orbiter warning: context deadline exceeded Duration: 3.46651342s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 6.468527511s trace: https://jaeger.dev.storj.io/trace/3c77028f820d12f9 FAIL (2025-11-16 13:45:09 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, Piece ID: UKP3MHXBSALCWFIHCTSR5JFFXEWWAWGDS4JYUYS2JEREQQKTPJSA) context deadline exceeded; piecestore: (Node ID: 12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, Piece ID: DNT5QVAMRO5OMBNVTDRRUKACMM7M5472LD5E4ASKYIBKBNWYAVRA) context deadline exceeded; piecestore: (Node ID: 17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, Piece ID: DPLALMJC4JPC5D7BOK5SOSDFDJFDDPGSJWP4XDDQE22AMFNAKATQ) context deadline exceeded ======LOGS====== gment. Inline: false Nov 16 13:42:52.666975419 (173h38m51.21844588s) [upload=152907]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 16 13:42:52.666980539 (173h38m51.21845099s) [upload=152907]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 16 13:42:53.024947049 (173h38m51.5764175s) [seg_pos={0 0}, upload=152907]: Begin upload segment... Nov 16 13:42:53.025007074 (173h38m51.576477535s) [seg_pos={0 0}, upload=152907]: Done begin upload segment. Nov 16 13:42:53.025087707 (173h38m51.576558158s) [noise=true , address=77.90.31.14:4662 , piece_id=S5PPMCKIKSOCLUJ4CEH7ZDW3TVUZ7GTFFEIPCQK6FX25UIOUQHHQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.025098377 (173h38m51.576568828s) [noise=true , address=145.79.227.31:4797 , piece_id=UKP3MHXBSALCWFIHCTSR5JFFXEWWAWGDS4JYUYS2JEREQQKTPJSA, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.025961451 (173h38m51.577431912s) [noise=true , address=45.91.116.6:4974 , piece_id=ZTS3H7E6A3VJECLRRKRKSVX5PH3VSH7F2HAUPQSPH56NCI6RTFBQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.026101888 (173h38m51.577572339s) [noise=true , address=145.79.227.10:4402 , piece_id=DNT5QVAMRO5OMBNVTDRRUKACMM7M5472LD5E4ASKYIBKBNWYAVRA, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.026397721 (173h38m51.577868182s) [noise=true , address=145.79.227.11:4732 , piece_id=GR6SASHNUTHAEOUPK7DWKVPHRGMG72FQMF4RINJ5NTIQQGFCVZZQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.026543650 (173h38m51.578014111s) [noise=true , address=145.79.227.20:4437 , piece_id=G5P6OYXVAAQDZKTPPHDNZLMYTT6L2TFWKH7N4F7L6742BWHOQGBQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.026887915 (173h38m51.578358366s) [noise=true , address=45.91.116.14:4189 , piece_id=NUMS6Q3F2OUAQ2PSKE2CZK2WKRQLZUZPWVBA3UA6UNKDQACPYSJQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.027039624 (173h38m51.578510085s) [noise=true , address=145.79.227.27:4691 , piece_id=NSX5S2D7ZSA5I57B4EAQEQXEVUMPKNMRKRV3RHWGWJ2DN7SWUPGQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.027374482 (173h38m51.578844973s) [noise=true , address=145.79.227.31:4798 , piece_id=DPLALMJC4JPC5D7BOK5SOSDFDJFDDPGSJWP4XDDQE22AMFNAKATQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.027472337 (173h38m51.578942798s) [noise=true , address=45.91.116.7:4010 , piece_id=EO7IEOCWEJAMZZU4BLIB5ALFACWTWZE3QCXU3FUY4FKC4P444ORA, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.027920821 (173h38m51.579391283s) [noise=true , address=145.79.227.23:4547 , piece_id=P4WX2ZDHV2Z6BFMF6IACIYW4DXKYFVSULUTF7EK2EMRO5OA5X5RQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.028194272 (173h38m51.579664743s) [noise=true , address=145.79.227.32:4834 , piece_id=CBK2UQ252EPFQDTOYS2X5WAUTJN4L2DTHGIOJXFZXWBOLGPRETXQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.028430262 (173h38m51.579900723s) [noise=true , address=145.79.227.9:4366 , piece_id=YFWF2YL2YPYVMP5DHJKRUGVUUS2ALG4DRJ3HUZCDZM4N3N676FNQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.029687726 (173h38m51.581158177s) [noise=true , address=145.79.227.20:4438 , piece_id=6HY2HDNJAZ552ZKXFJ7VNTQNBAQU5ZQLNKAG2NZZWZQHN6EEDG6Q, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.031784128 (173h38m51.583254589s) [noise=true , address=45.91.116.5:4298 , piece_id=ARGPAZE6CTMM6UXSMPSVHDBEQWKVGKY7WXOVLPB7UH3JZYMDB7JQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.034100830 (173h38m51.585571291s) [noise=true , address=77.90.31.10:4518 , piece_id=MSKQ66KE4WANAE4AWOD2IKY3PQN2YGZ5N4W3NXGKXSOLE7B6H6WQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:54.850200282 (173h38m53.401670733s) [upload=152907]: Next returned nil segment Nov 16 13:42:54.850219118 (173h38m53.401689569s) [upload=152907]: Waiting for error group managing segments... Nov 16 13:42:55.154333032 (173h38m53.705803493s) [noise=true , address=45.91.116.7:4010 , piece_id=EO7IEOCWEJAMZZU4BLIB5ALFACWTWZE3QCXU3FUY4FKC4P444ORA, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:42:55.201689040 (173h38m53.753159501s) [noise=true , address=77.90.31.10:4518 , piece_id=MSKQ66KE4WANAE4AWOD2IKY3PQN2YGZ5N4W3NXGKXSOLE7B6H6WQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:42:55.232313345 (173h38m53.783783816s) [noise=true , address=45.91.116.5:4298 , piece_id=ARGPAZE6CTMM6UXSMPSVHDBEQWKVGKY7WXOVLPB7UH3JZYMDB7JQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:42:55.260281393 (173h38m53.811751854s) [noise=true , address=45.91.116.6:4974 , piece_id=ZTS3H7E6A3VJECLRRKRKSVX5PH3VSH7F2HAUPQSPH56NCI6RTFBQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:00.229127085 (173h38m58.780597546s) [noise=true , address=145.79.227.20:4438 , piece_id=6HY2HDNJAZ552ZKXFJ7VNTQNBAQU5ZQLNKAG2NZZWZQHN6EEDG6Q, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:00.358241119 (173h38m58.909711581s) [noise=true , address=145.79.227.32:4834 , piece_id=CBK2UQ252EPFQDTOYS2X5WAUTJN4L2DTHGIOJXFZXWBOLGPRETXQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:00.646958215 (173h38m59.198428676s) [noise=true , address=145.79.227.31:4798 , piece_id=DPLALMJC4JPC5D7BOK5SOSDFDJFDDPGSJWP4XDDQE22AMFNAKATQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:00.908824080 (173h38m59.460294541s) [noise=true , address=145.79.227.23:4547 , piece_id=P4WX2ZDHV2Z6BFMF6IACIYW4DXKYFVSULUTF7EK2EMRO5OA5X5RQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:01.155346527 (173h38m59.706816989s) [noise=true , address=45.91.116.14:4189 , piece_id=NUMS6Q3F2OUAQ2PSKE2CZK2WKRQLZUZPWVBA3UA6UNKDQACPYSJQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:01.839444794 (173h39m0.390915255s) [noise=true , address=145.79.227.11:4732 , piece_id=GR6SASHNUTHAEOUPK7DWKVPHRGMG72FQMF4RINJ5NTIQQGFCVZZQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:02.591249579 (173h39m1.14272004s) [noise=true , address=145.79.227.10:4402 , piece_id=DNT5QVAMRO5OMBNVTDRRUKACMM7M5472LD5E4ASKYIBKBNWYAVRA, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:02.616126740 (173h39m1.167597202s) [noise=true , address=145.79.227.31:4797 , piece_id=UKP3MHXBSALCWFIHCTSR5JFFXEWWAWGDS4JYUYS2JEREQQKTPJSA, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:02.734605482 (173h39m1.286075943s) [noise=true , address=145.79.227.27:4691 , piece_id=NSX5S2D7ZSA5I57B4EAQEQXEVUMPKNMRKRV3RHWGWJ2DN7SWUPGQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:02.734638004 (173h39m1.286108475s) [seg_pos={0 0}, upload=152907]: Segment reached optimal threshold of 13 pieces. Nov 16 13:43:02.735049748 (173h39m1.286520209s) [noise=true , address=145.79.227.9:4366 , piece_id=YFWF2YL2YPYVMP5DHJKRUGVUUS2ALG4DRJ3HUZCDZM4N3N676FNQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy): protocol: write tcp 188.166.225.170:60100->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:60100->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60100->145.79.227.9:4366: use of closed network connection Nov 16 13:43:02.735060949 (173h39m1.28653141s) [noise=true , address=145.79.227.9:4366 , piece_id=YFWF2YL2YPYVMP5DHJKRUGVUUS2ALG4DRJ3HUZCDZM4N3N676FNQ, seg_pos={0 0}, upload=152907]: Long tail cancellation due to optimal threshold Nov 16 13:43:02.735509193 (173h39m1.286979654s) [noise=true , address=77.90.31.14:4662 , piece_id=S5PPMCKIKSOCLUJ4CEH7ZDW3TVUZ7GTFFEIPCQK6FX25UIOUQHHQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): protocol: write tcp 188.166.225.170:50196->77.90.31.14:4662: use of closed network connection; write tcp 188.166.225.170:50196->77.90.31.14:4662: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50196->77.90.31.14:4662: use of closed network connection Nov 16 13:43:02.735518792 (173h39m1.286989253s) [noise=true , address=77.90.31.14:4662 , piece_id=S5PPMCKIKSOCLUJ4CEH7ZDW3TVUZ7GTFFEIPCQK6FX25UIOUQHHQ, seg_pos={0 0}, upload=152907]: Long tail cancellation due to optimal threshold Nov 16 13:43:02.736224335 (173h39m1.287694796s) [noise=true , address=145.79.227.20:4437 , piece_id=G5P6OYXVAAQDZKTPPHDNZLMYTT6L2TFWKH7N4F7L6742BWHOQGBQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV): protocol: read tcp 188.166.225.170:35300->145.79.227.20:4437: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 13:43:02.736239203 (173h39m1.287709654s) [noise=true , address=145.79.227.20:4437 , piece_id=G5P6OYXVAAQDZKTPPHDNZLMYTT6L2TFWKH7N4F7L6742BWHOQGBQ, seg_pos={0 0}, upload=152907]: Long tail cancellation due to optimal threshold Nov 16 13:43:02.736270833 (173h39m1.287741294s) [seg_pos={0 0}, upload=152907]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 16 13:43:02.736437060 (173h39m1.287907521s) [upload=152907]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 16 13:43:02.736444273 (173h39m1.287914735s) [upload=152907]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 16 13:43:05.031396865 (173h39m3.582867336s) [upload=152907]: Done uploading segments... Nov 16 13:43:05.031422193 (173h39m3.582892654s) [upload=152907]: Upload finished. err: <nil> Duration: 6.46899449s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 11 out of 60 is failed (threshold=6) [X................XX....X........X.........XX....X......X..XX] duration: 5.001711871s trace: https://jaeger.dev.storj.io/trace/e4f9648cfff91e2 FAIL (2025-11-16 13:44:58 UTC) orbiter warning: context deadline exceeded Duration: 5.00173796s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 11 out of 60 is failed (threshold=6) [XX................XX....X........X.........XX....X......X..X] duration: 5.001744853s trace: https://jaeger.dev.storj.io/trace/52431a4d2b32424 FAIL (2025-11-16 13:44:38 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001767616s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 14.440642289s trace: https://jaeger.dev.storj.io/trace/129f27707330b16c FAIL (2025-11-16 13:44:27 UTC) orbiter warning: context deadline exceeded Duration: 14.440661485s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 10 out of 60 is failed (threshold=6) [.XX................XX....X........X.........XX....X......X..] duration: 3.445257313s trace: https://jaeger.dev.storj.io/trace/dccc4dd84f5498c FAIL (2025-11-16 13:44:00 UTC) orbiter warning: context deadline exceeded Duration: 3.445284916s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 6.307313773s trace: https://jaeger.dev.storj.io/trace/3c77028f820d12f9 FAIL (2025-11-16 13:43:57 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, Piece ID: UKP3MHXBSALCWFIHCTSR5JFFXEWWAWGDS4JYUYS2JEREQQKTPJSA) context deadline exceeded; piecestore: (Node ID: 12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, Piece ID: DNT5QVAMRO5OMBNVTDRRUKACMM7M5472LD5E4ASKYIBKBNWYAVRA) context deadline exceeded; piecestore: (Node ID: 17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, Piece ID: DPLALMJC4JPC5D7BOK5SOSDFDJFDDPGSJWP4XDDQE22AMFNAKATQ) context deadline exceeded ======LOGS====== gment. Inline: false Nov 16 13:42:52.666975419 (173h38m51.21844588s) [upload=152907]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 16 13:42:52.666980539 (173h38m51.21845099s) [upload=152907]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 16 13:42:53.024947049 (173h38m51.5764175s) [seg_pos={0 0}, upload=152907]: Begin upload segment... Nov 16 13:42:53.025007074 (173h38m51.576477535s) [seg_pos={0 0}, upload=152907]: Done begin upload segment. Nov 16 13:42:53.025087707 (173h38m51.576558158s) [noise=true , address=77.90.31.14:4662 , piece_id=S5PPMCKIKSOCLUJ4CEH7ZDW3TVUZ7GTFFEIPCQK6FX25UIOUQHHQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.025098377 (173h38m51.576568828s) [noise=true , address=145.79.227.31:4797 , piece_id=UKP3MHXBSALCWFIHCTSR5JFFXEWWAWGDS4JYUYS2JEREQQKTPJSA, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.025961451 (173h38m51.577431912s) [noise=true , address=45.91.116.6:4974 , piece_id=ZTS3H7E6A3VJECLRRKRKSVX5PH3VSH7F2HAUPQSPH56NCI6RTFBQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.026101888 (173h38m51.577572339s) [noise=true , address=145.79.227.10:4402 , piece_id=DNT5QVAMRO5OMBNVTDRRUKACMM7M5472LD5E4ASKYIBKBNWYAVRA, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.026397721 (173h38m51.577868182s) [noise=true , address=145.79.227.11:4732 , piece_id=GR6SASHNUTHAEOUPK7DWKVPHRGMG72FQMF4RINJ5NTIQQGFCVZZQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.026543650 (173h38m51.578014111s) [noise=true , address=145.79.227.20:4437 , piece_id=G5P6OYXVAAQDZKTPPHDNZLMYTT6L2TFWKH7N4F7L6742BWHOQGBQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.026887915 (173h38m51.578358366s) [noise=true , address=45.91.116.14:4189 , piece_id=NUMS6Q3F2OUAQ2PSKE2CZK2WKRQLZUZPWVBA3UA6UNKDQACPYSJQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.027039624 (173h38m51.578510085s) [noise=true , address=145.79.227.27:4691 , piece_id=NSX5S2D7ZSA5I57B4EAQEQXEVUMPKNMRKRV3RHWGWJ2DN7SWUPGQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.027374482 (173h38m51.578844973s) [noise=true , address=145.79.227.31:4798 , piece_id=DPLALMJC4JPC5D7BOK5SOSDFDJFDDPGSJWP4XDDQE22AMFNAKATQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.027472337 (173h38m51.578942798s) [noise=true , address=45.91.116.7:4010 , piece_id=EO7IEOCWEJAMZZU4BLIB5ALFACWTWZE3QCXU3FUY4FKC4P444ORA, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.027920821 (173h38m51.579391283s) [noise=true , address=145.79.227.23:4547 , piece_id=P4WX2ZDHV2Z6BFMF6IACIYW4DXKYFVSULUTF7EK2EMRO5OA5X5RQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.028194272 (173h38m51.579664743s) [noise=true , address=145.79.227.32:4834 , piece_id=CBK2UQ252EPFQDTOYS2X5WAUTJN4L2DTHGIOJXFZXWBOLGPRETXQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.028430262 (173h38m51.579900723s) [noise=true , address=145.79.227.9:4366 , piece_id=YFWF2YL2YPYVMP5DHJKRUGVUUS2ALG4DRJ3HUZCDZM4N3N676FNQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.029687726 (173h38m51.581158177s) [noise=true , address=145.79.227.20:4438 , piece_id=6HY2HDNJAZ552ZKXFJ7VNTQNBAQU5ZQLNKAG2NZZWZQHN6EEDG6Q, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.031784128 (173h38m51.583254589s) [noise=true , address=45.91.116.5:4298 , piece_id=ARGPAZE6CTMM6UXSMPSVHDBEQWKVGKY7WXOVLPB7UH3JZYMDB7JQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.034100830 (173h38m51.585571291s) [noise=true , address=77.90.31.10:4518 , piece_id=MSKQ66KE4WANAE4AWOD2IKY3PQN2YGZ5N4W3NXGKXSOLE7B6H6WQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:54.850200282 (173h38m53.401670733s) [upload=152907]: Next returned nil segment Nov 16 13:42:54.850219118 (173h38m53.401689569s) [upload=152907]: Waiting for error group managing segments... Nov 16 13:42:55.154333032 (173h38m53.705803493s) [noise=true , address=45.91.116.7:4010 , piece_id=EO7IEOCWEJAMZZU4BLIB5ALFACWTWZE3QCXU3FUY4FKC4P444ORA, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:42:55.201689040 (173h38m53.753159501s) [noise=true , address=77.90.31.10:4518 , piece_id=MSKQ66KE4WANAE4AWOD2IKY3PQN2YGZ5N4W3NXGKXSOLE7B6H6WQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:42:55.232313345 (173h38m53.783783816s) [noise=true , address=45.91.116.5:4298 , piece_id=ARGPAZE6CTMM6UXSMPSVHDBEQWKVGKY7WXOVLPB7UH3JZYMDB7JQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:42:55.260281393 (173h38m53.811751854s) [noise=true , address=45.91.116.6:4974 , piece_id=ZTS3H7E6A3VJECLRRKRKSVX5PH3VSH7F2HAUPQSPH56NCI6RTFBQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:00.229127085 (173h38m58.780597546s) [noise=true , address=145.79.227.20:4438 , piece_id=6HY2HDNJAZ552ZKXFJ7VNTQNBAQU5ZQLNKAG2NZZWZQHN6EEDG6Q, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:00.358241119 (173h38m58.909711581s) [noise=true , address=145.79.227.32:4834 , piece_id=CBK2UQ252EPFQDTOYS2X5WAUTJN4L2DTHGIOJXFZXWBOLGPRETXQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:00.646958215 (173h38m59.198428676s) [noise=true , address=145.79.227.31:4798 , piece_id=DPLALMJC4JPC5D7BOK5SOSDFDJFDDPGSJWP4XDDQE22AMFNAKATQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:00.908824080 (173h38m59.460294541s) [noise=true , address=145.79.227.23:4547 , piece_id=P4WX2ZDHV2Z6BFMF6IACIYW4DXKYFVSULUTF7EK2EMRO5OA5X5RQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:01.155346527 (173h38m59.706816989s) [noise=true , address=45.91.116.14:4189 , piece_id=NUMS6Q3F2OUAQ2PSKE2CZK2WKRQLZUZPWVBA3UA6UNKDQACPYSJQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:01.839444794 (173h39m0.390915255s) [noise=true , address=145.79.227.11:4732 , piece_id=GR6SASHNUTHAEOUPK7DWKVPHRGMG72FQMF4RINJ5NTIQQGFCVZZQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:02.591249579 (173h39m1.14272004s) [noise=true , address=145.79.227.10:4402 , piece_id=DNT5QVAMRO5OMBNVTDRRUKACMM7M5472LD5E4ASKYIBKBNWYAVRA, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:02.616126740 (173h39m1.167597202s) [noise=true , address=145.79.227.31:4797 , piece_id=UKP3MHXBSALCWFIHCTSR5JFFXEWWAWGDS4JYUYS2JEREQQKTPJSA, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:02.734605482 (173h39m1.286075943s) [noise=true , address=145.79.227.27:4691 , piece_id=NSX5S2D7ZSA5I57B4EAQEQXEVUMPKNMRKRV3RHWGWJ2DN7SWUPGQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:02.734638004 (173h39m1.286108475s) [seg_pos={0 0}, upload=152907]: Segment reached optimal threshold of 13 pieces. Nov 16 13:43:02.735049748 (173h39m1.286520209s) [noise=true , address=145.79.227.9:4366 , piece_id=YFWF2YL2YPYVMP5DHJKRUGVUUS2ALG4DRJ3HUZCDZM4N3N676FNQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy): protocol: write tcp 188.166.225.170:60100->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:60100->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60100->145.79.227.9:4366: use of closed network connection Nov 16 13:43:02.735060949 (173h39m1.28653141s) [noise=true , address=145.79.227.9:4366 , piece_id=YFWF2YL2YPYVMP5DHJKRUGVUUS2ALG4DRJ3HUZCDZM4N3N676FNQ, seg_pos={0 0}, upload=152907]: Long tail cancellation due to optimal threshold Nov 16 13:43:02.735509193 (173h39m1.286979654s) [noise=true , address=77.90.31.14:4662 , piece_id=S5PPMCKIKSOCLUJ4CEH7ZDW3TVUZ7GTFFEIPCQK6FX25UIOUQHHQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): protocol: write tcp 188.166.225.170:50196->77.90.31.14:4662: use of closed network connection; write tcp 188.166.225.170:50196->77.90.31.14:4662: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50196->77.90.31.14:4662: use of closed network connection Nov 16 13:43:02.735518792 (173h39m1.286989253s) [noise=true , address=77.90.31.14:4662 , piece_id=S5PPMCKIKSOCLUJ4CEH7ZDW3TVUZ7GTFFEIPCQK6FX25UIOUQHHQ, seg_pos={0 0}, upload=152907]: Long tail cancellation due to optimal threshold Nov 16 13:43:02.736224335 (173h39m1.287694796s) [noise=true , address=145.79.227.20:4437 , piece_id=G5P6OYXVAAQDZKTPPHDNZLMYTT6L2TFWKH7N4F7L6742BWHOQGBQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV): protocol: read tcp 188.166.225.170:35300->145.79.227.20:4437: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 13:43:02.736239203 (173h39m1.287709654s) [noise=true , address=145.79.227.20:4437 , piece_id=G5P6OYXVAAQDZKTPPHDNZLMYTT6L2TFWKH7N4F7L6742BWHOQGBQ, seg_pos={0 0}, upload=152907]: Long tail cancellation due to optimal threshold Nov 16 13:43:02.736270833 (173h39m1.287741294s) [seg_pos={0 0}, upload=152907]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 16 13:43:02.736437060 (173h39m1.287907521s) [upload=152907]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 16 13:43:02.736444273 (173h39m1.287914735s) [upload=152907]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 16 13:43:05.031396865 (173h39m3.582867336s) [upload=152907]: Done uploading segments... Nov 16 13:43:05.031422193 (173h39m3.582892654s) [upload=152907]: Upload finished. err: <nil> Duration: 6.307346655s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.003541206s trace: https://jaeger.dev.storj.io/trace/3c77028f820d12f9 FAIL (2025-11-16 13:43:20 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, Piece ID: UKP3MHXBSALCWFIHCTSR5JFFXEWWAWGDS4JYUYS2JEREQQKTPJSA) context deadline exceeded; piecestore: (Node ID: 12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, Piece ID: DNT5QVAMRO5OMBNVTDRRUKACMM7M5472LD5E4ASKYIBKBNWYAVRA) context deadline exceeded; piecestore: (Node ID: 17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, Piece ID: DPLALMJC4JPC5D7BOK5SOSDFDJFDDPGSJWP4XDDQE22AMFNAKATQ) context deadline exceeded ======LOGS====== gment. Inline: false Nov 16 13:42:52.666975419 (173h38m51.21844588s) [upload=152907]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 16 13:42:52.666980539 (173h38m51.21845099s) [upload=152907]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 16 13:42:53.024947049 (173h38m51.5764175s) [seg_pos={0 0}, upload=152907]: Begin upload segment... Nov 16 13:42:53.025007074 (173h38m51.576477535s) [seg_pos={0 0}, upload=152907]: Done begin upload segment. Nov 16 13:42:53.025087707 (173h38m51.576558158s) [noise=true , address=77.90.31.14:4662 , piece_id=S5PPMCKIKSOCLUJ4CEH7ZDW3TVUZ7GTFFEIPCQK6FX25UIOUQHHQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.025098377 (173h38m51.576568828s) [noise=true , address=145.79.227.31:4797 , piece_id=UKP3MHXBSALCWFIHCTSR5JFFXEWWAWGDS4JYUYS2JEREQQKTPJSA, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.025961451 (173h38m51.577431912s) [noise=true , address=45.91.116.6:4974 , piece_id=ZTS3H7E6A3VJECLRRKRKSVX5PH3VSH7F2HAUPQSPH56NCI6RTFBQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.026101888 (173h38m51.577572339s) [noise=true , address=145.79.227.10:4402 , piece_id=DNT5QVAMRO5OMBNVTDRRUKACMM7M5472LD5E4ASKYIBKBNWYAVRA, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.026397721 (173h38m51.577868182s) [noise=true , address=145.79.227.11:4732 , piece_id=GR6SASHNUTHAEOUPK7DWKVPHRGMG72FQMF4RINJ5NTIQQGFCVZZQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.026543650 (173h38m51.578014111s) [noise=true , address=145.79.227.20:4437 , piece_id=G5P6OYXVAAQDZKTPPHDNZLMYTT6L2TFWKH7N4F7L6742BWHOQGBQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.026887915 (173h38m51.578358366s) [noise=true , address=45.91.116.14:4189 , piece_id=NUMS6Q3F2OUAQ2PSKE2CZK2WKRQLZUZPWVBA3UA6UNKDQACPYSJQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.027039624 (173h38m51.578510085s) [noise=true , address=145.79.227.27:4691 , piece_id=NSX5S2D7ZSA5I57B4EAQEQXEVUMPKNMRKRV3RHWGWJ2DN7SWUPGQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.027374482 (173h38m51.578844973s) [noise=true , address=145.79.227.31:4798 , piece_id=DPLALMJC4JPC5D7BOK5SOSDFDJFDDPGSJWP4XDDQE22AMFNAKATQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.027472337 (173h38m51.578942798s) [noise=true , address=45.91.116.7:4010 , piece_id=EO7IEOCWEJAMZZU4BLIB5ALFACWTWZE3QCXU3FUY4FKC4P444ORA, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.027920821 (173h38m51.579391283s) [noise=true , address=145.79.227.23:4547 , piece_id=P4WX2ZDHV2Z6BFMF6IACIYW4DXKYFVSULUTF7EK2EMRO5OA5X5RQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.028194272 (173h38m51.579664743s) [noise=true , address=145.79.227.32:4834 , piece_id=CBK2UQ252EPFQDTOYS2X5WAUTJN4L2DTHGIOJXFZXWBOLGPRETXQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.028430262 (173h38m51.579900723s) [noise=true , address=145.79.227.9:4366 , piece_id=YFWF2YL2YPYVMP5DHJKRUGVUUS2ALG4DRJ3HUZCDZM4N3N676FNQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.029687726 (173h38m51.581158177s) [noise=true , address=145.79.227.20:4438 , piece_id=6HY2HDNJAZ552ZKXFJ7VNTQNBAQU5ZQLNKAG2NZZWZQHN6EEDG6Q, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.031784128 (173h38m51.583254589s) [noise=true , address=45.91.116.5:4298 , piece_id=ARGPAZE6CTMM6UXSMPSVHDBEQWKVGKY7WXOVLPB7UH3JZYMDB7JQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:53.034100830 (173h38m51.585571291s) [noise=true , address=77.90.31.10:4518 , piece_id=MSKQ66KE4WANAE4AWOD2IKY3PQN2YGZ5N4W3NXGKXSOLE7B6H6WQ, seg_pos={0 0}, upload=152907]: Uploading piece... Nov 16 13:42:54.850200282 (173h38m53.401670733s) [upload=152907]: Next returned nil segment Nov 16 13:42:54.850219118 (173h38m53.401689569s) [upload=152907]: Waiting for error group managing segments... Nov 16 13:42:55.154333032 (173h38m53.705803493s) [noise=true , address=45.91.116.7:4010 , piece_id=EO7IEOCWEJAMZZU4BLIB5ALFACWTWZE3QCXU3FUY4FKC4P444ORA, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:42:55.201689040 (173h38m53.753159501s) [noise=true , address=77.90.31.10:4518 , piece_id=MSKQ66KE4WANAE4AWOD2IKY3PQN2YGZ5N4W3NXGKXSOLE7B6H6WQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:42:55.232313345 (173h38m53.783783816s) [noise=true , address=45.91.116.5:4298 , piece_id=ARGPAZE6CTMM6UXSMPSVHDBEQWKVGKY7WXOVLPB7UH3JZYMDB7JQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:42:55.260281393 (173h38m53.811751854s) [noise=true , address=45.91.116.6:4974 , piece_id=ZTS3H7E6A3VJECLRRKRKSVX5PH3VSH7F2HAUPQSPH56NCI6RTFBQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:00.229127085 (173h38m58.780597546s) [noise=true , address=145.79.227.20:4438 , piece_id=6HY2HDNJAZ552ZKXFJ7VNTQNBAQU5ZQLNKAG2NZZWZQHN6EEDG6Q, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:00.358241119 (173h38m58.909711581s) [noise=true , address=145.79.227.32:4834 , piece_id=CBK2UQ252EPFQDTOYS2X5WAUTJN4L2DTHGIOJXFZXWBOLGPRETXQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:00.646958215 (173h38m59.198428676s) [noise=true , address=145.79.227.31:4798 , piece_id=DPLALMJC4JPC5D7BOK5SOSDFDJFDDPGSJWP4XDDQE22AMFNAKATQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:00.908824080 (173h38m59.460294541s) [noise=true , address=145.79.227.23:4547 , piece_id=P4WX2ZDHV2Z6BFMF6IACIYW4DXKYFVSULUTF7EK2EMRO5OA5X5RQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:01.155346527 (173h38m59.706816989s) [noise=true , address=45.91.116.14:4189 , piece_id=NUMS6Q3F2OUAQ2PSKE2CZK2WKRQLZUZPWVBA3UA6UNKDQACPYSJQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:01.839444794 (173h39m0.390915255s) [noise=true , address=145.79.227.11:4732 , piece_id=GR6SASHNUTHAEOUPK7DWKVPHRGMG72FQMF4RINJ5NTIQQGFCVZZQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:02.591249579 (173h39m1.14272004s) [noise=true , address=145.79.227.10:4402 , piece_id=DNT5QVAMRO5OMBNVTDRRUKACMM7M5472LD5E4ASKYIBKBNWYAVRA, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:02.616126740 (173h39m1.167597202s) [noise=true , address=145.79.227.31:4797 , piece_id=UKP3MHXBSALCWFIHCTSR5JFFXEWWAWGDS4JYUYS2JEREQQKTPJSA, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:02.734605482 (173h39m1.286075943s) [noise=true , address=145.79.227.27:4691 , piece_id=NSX5S2D7ZSA5I57B4EAQEQXEVUMPKNMRKRV3RHWGWJ2DN7SWUPGQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: <nil> Nov 16 13:43:02.734638004 (173h39m1.286108475s) [seg_pos={0 0}, upload=152907]: Segment reached optimal threshold of 13 pieces. Nov 16 13:43:02.735049748 (173h39m1.286520209s) [noise=true , address=145.79.227.9:4366 , piece_id=YFWF2YL2YPYVMP5DHJKRUGVUUS2ALG4DRJ3HUZCDZM4N3N676FNQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy): protocol: write tcp 188.166.225.170:60100->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:60100->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60100->145.79.227.9:4366: use of closed network connection Nov 16 13:43:02.735060949 (173h39m1.28653141s) [noise=true , address=145.79.227.9:4366 , piece_id=YFWF2YL2YPYVMP5DHJKRUGVUUS2ALG4DRJ3HUZCDZM4N3N676FNQ, seg_pos={0 0}, upload=152907]: Long tail cancellation due to optimal threshold Nov 16 13:43:02.735509193 (173h39m1.286979654s) [noise=true , address=77.90.31.14:4662 , piece_id=S5PPMCKIKSOCLUJ4CEH7ZDW3TVUZ7GTFFEIPCQK6FX25UIOUQHHQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): protocol: write tcp 188.166.225.170:50196->77.90.31.14:4662: use of closed network connection; write tcp 188.166.225.170:50196->77.90.31.14:4662: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50196->77.90.31.14:4662: use of closed network connection Nov 16 13:43:02.735518792 (173h39m1.286989253s) [noise=true , address=77.90.31.14:4662 , piece_id=S5PPMCKIKSOCLUJ4CEH7ZDW3TVUZ7GTFFEIPCQK6FX25UIOUQHHQ, seg_pos={0 0}, upload=152907]: Long tail cancellation due to optimal threshold Nov 16 13:43:02.736224335 (173h39m1.287694796s) [noise=true , address=145.79.227.20:4437 , piece_id=G5P6OYXVAAQDZKTPPHDNZLMYTT6L2TFWKH7N4F7L6742BWHOQGBQ, seg_pos={0 0}, upload=152907]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV): protocol: read tcp 188.166.225.170:35300->145.79.227.20:4437: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 16 13:43:02.736239203 (173h39m1.287709654s) [noise=true , address=145.79.227.20:4437 , piece_id=G5P6OYXVAAQDZKTPPHDNZLMYTT6L2TFWKH7N4F7L6742BWHOQGBQ, seg_pos={0 0}, upload=152907]: Long tail cancellation due to optimal threshold Nov 16 13:43:02.736270833 (173h39m1.287741294s) [seg_pos={0 0}, upload=152907]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 16 13:43:02.736437060 (173h39m1.287907521s) [upload=152907]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 16 13:43:02.736444273 (173h39m1.287914735s) [upload=152907]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 16 13:43:05.031396865 (173h39m3.582867336s) [upload=152907]: Done uploading segments... Nov 16 13:43:05.031422193 (173h39m3.582892654s) [upload=152907]: Upload finished. err: <nil> Duration: 30.003567506s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 7.93656544s trace: https://jaeger.dev.storj.io/trace/129f27707330b16c FAIL (2025-11-16 13:43:07 UTC) orbiter warning: context deadline exceeded Duration: 7.936891371s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 10 out of 60 is failed (threshold=6) [..XX................XX....X........X.........XX....X......X.] duration: 3.773071786s trace: https://jaeger.dev.storj.io/trace/dccc4dd84f5498c FAIL (2025-11-16 13:42:49 UTC) orbiter warning: context deadline exceeded Duration: 3.773098076s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 10 out of 60 is failed (threshold=6) [...XX................XX....X........X.........XX....X......X] duration: 5.001696971s trace: https://jaeger.dev.storj.io/trace/dccc4dd84f5498c FAIL (2025-11-16 13:42:18 UTC) orbiter warning: context deadline exceeded Duration: 5.001746385s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 6.495834724s trace: https://jaeger.dev.storj.io/trace/129f27707330b16c FAIL (2025-11-16 13:42:14 UTC) orbiter warning: context deadline exceeded Duration: 6.496174571s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 11.453415852s trace: https://jaeger.dev.storj.io/trace/129f27707330b16c FAIL (2025-11-16 13:41:40 UTC) orbiter warning: context deadline exceeded Duration: 11.453483703s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 9 out of 60 is failed (threshold=6) [....XX................XX....X........X.........XX....X......] duration: 3.563038399s trace: https://jaeger.dev.storj.io/trace/362c70670c2feb08 FAIL (2025-11-16 13:41:16 UTC) orbiter warning: context deadline exceeded Duration: 3.563062175s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [X.....................................X.....................] duration: 10.79322151s trace: https://jaeger.dev.storj.io/trace/129f27707330b16c FAIL (2025-11-16 13:41:14 UTC) orbiter warning: context deadline exceeded Duration: 10.793253711s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.X.....................................X....................] duration: 9.232875286s trace: https://jaeger.dev.storj.io/trace/129f27707330b16c FAIL (2025-11-16 13:40:51 UTC) orbiter warning: context deadline exceeded Duration: 9.23289832s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 9 out of 60 is failed (threshold=6) [.....XX................XX....X........X.........XX....X.....] duration: 4.619502442s trace: https://jaeger.dev.storj.io/trace/362c70670c2feb08 FAIL (2025-11-16 13:40:45 UTC) orbiter warning: context deadline exceeded Duration: 4.619529774s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..X.....................................X...................] duration: 7.605841151s trace: https://jaeger.dev.storj.io/trace/129f27707330b16c FAIL (2025-11-16 13:40:23 UTC) orbiter warning: context deadline exceeded Duration: 7.605879093s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 9 out of 60 is failed (threshold=6) [......XX................XX....X........X.........XX....X....] duration: 3.534207709s trace: https://jaeger.dev.storj.io/trace/362c70670c2feb08 FAIL (2025-11-16 13:39:48 UTC) orbiter warning: context deadline exceeded Duration: 3.534244138s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...X.....................................X..................] duration: 9.93804191s trace: https://jaeger.dev.storj.io/trace/129f27707330b16c FAIL (2025-11-16 13:39:24 UTC) orbiter warning: context deadline exceeded Duration: 9.938059883s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [....X.....................................X.................] duration: 10.734642407s trace: https://jaeger.dev.storj.io/trace/129f27707330b16c FAIL (2025-11-16 13:39:04 UTC) orbiter warning: context deadline exceeded Duration: 10.734673916s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 10 out of 60 is failed (threshold=6) [X......XX................XX....X........X.........XX....X...] duration: 4.113465858s trace: https://jaeger.dev.storj.io/trace/362c70670c2feb08 FAIL (2025-11-16 13:39:03 UTC) orbiter warning: context deadline exceeded Duration: 4.113495535s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....X.....................................X................] duration: 12.451196859s trace: https://jaeger.dev.storj.io/trace/129f27707330b16c FAIL (2025-11-16 13:38:49 UTC) orbiter warning: context deadline exceeded Duration: 12.451292923s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 10 out of 60 is failed (threshold=6) [.X......XX................XX....X........X.........XX....X..] duration: 3.728705064s trace: https://jaeger.dev.storj.io/trace/362c70670c2feb08 FAIL (2025-11-16 13:38:20 UTC) orbiter warning: context deadline exceeded Duration: 3.728732837s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [......X.....................................X...............] duration: 7.395621454s trace: https://jaeger.dev.storj.io/trace/129f27707330b16c FAIL (2025-11-16 13:38:16 UTC) orbiter warning: context deadline exceeded Duration: 7.395649917s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 10 out of 60 is failed (threshold=6) [..X......XX................XX....X........X.........XX....X.] duration: 5.000753156s trace: https://jaeger.dev.storj.io/trace/362c70670c2feb08 FAIL (2025-11-16 13:37:56 UTC) orbiter warning: context deadline exceeded Duration: 5.000774538s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 10 out of 60 is failed (threshold=6) [...X......XX................XX....X........X.........XX....X] duration: 5.00122385s trace: https://jaeger.dev.storj.io/trace/362c70670c2feb08 FAIL (2025-11-16 13:37:31 UTC) orbiter warning: context deadline exceeded Duration: 5.001248156s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......X.....................................X..............] duration: 10.83939271s trace: https://jaeger.dev.storj.io/trace/129f27707330b16c FAIL (2025-11-16 13:37:25 UTC) orbiter warning: context deadline exceeded Duration: 10.839424779s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [........X.....................................X.............] duration: 18.003687252s trace: https://jaeger.dev.storj.io/trace/129f27707330b16c FAIL (2025-11-16 13:36:51 UTC) orbiter warning: context deadline exceeded Duration: 18.003714846s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 9 out of 60 is failed (threshold=6) [....X......XX................XX....X........X.........XX....] duration: 3.497896584s trace: https://jaeger.dev.storj.io/trace/52bf8698ed9ed6c5 FAIL (2025-11-16 13:36:21 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.49791534s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)