Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [........X...................................................] duration: 3.531332134s trace: https://jaeger.dev.storj.io/trace/1f3e769edbb82b7a FAIL (2025-12-04 17:40:39 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.531362691s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 5.73333874s trace: https://jaeger.dev.storj.io/trace/6d15aafc5ce7b199 FAIL (2025-12-04 17:40:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, Piece ID: FBD34B6US36TX6YHN5UEBKOWIVOOP4G7B6SSYKN6Q3J5YNE7OBHA) context deadline exceeded; piecestore: (Node ID: 12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9, Piece ID: ULLE3QB2QSCSYQWZ4OSBDJHC3X7AOHJGRWRA4DUXAMOYBDJRYWRQ) context deadline exceeded; piecestore: (Node ID: 1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, Piece ID: TLP7B6MNA7W35UJQKW5SW6F43YXXMUFXOUKW4DKSOWOPIKLSSVUA) context deadline exceeded; piecestore: (Node ID: 17o881pUMuG7K2dv83k7RkrJCrmM7kgUiWesgam4GEw21KVUn7, Piece ID: QOOXF6G2OGBNJLA64B2YNVVJKAZSP6HM5ZIEI4ZFJKWZUSQAR3XA) context deadline exceeded; piecestore: (Node ID: 1ZrzADk2mKG4R9tjxUKjQeJvb4WmfjLksGFyvstHRfYbBC7XNz, Piece ID: VWS32XVNWH4R5PZDDV4FGP5ONZQZNLMU5YYB7QYS43GDJSDFX2OQ) context deadline exceeded ======LOGS====== Dec 4 17:22:34.320746327 (24h27m10.791957066s) [upload=21507]: Starting upload Dec 4 17:22:34.320860715 (24h27m10.792071454s) [upload=21507]: Done starting upload Dec 4 17:22:34.321208968 (24h27m10.792419707s) [upload=21507]: Uploading segments... Dec 4 17:22:34.321325259 (24h27m10.792536018s) [upload=21507]: Got next segment. Inline: false Dec 4 17:22:34.321394982 (24h27m10.792605711s) [upload=21507]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 17:22:34.321401375 (24h27m10.792612114s) [upload=21507]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 17:22:34.827776318 (24h27m11.298987057s) [seg_pos={0 0}, upload=21507]: Begin upload segment... Dec 4 17:22:34.827838226 (24h27m11.299048966s) [seg_pos={0 0}, upload=21507]: Done begin upload segment. Dec 4 17:22:34.827877361 (24h27m11.29908811s) [noise=true , address=77.90.31.14:4661 , piece_id=D5L5WWE5DGEYKP5M2NM74EGSX5BLEP4DX5OBM2Z4Q2QTCSGA4WXQ, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.827887630 (24h27m11.299098359s) [noise=true , address=77.90.31.21:4878 , piece_id=GHU3SZUB5WAMW37CYCUFJ6HOPMXRNZGIJYDC4DJ3WOHV7YYWCVEA, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.828727861 (24h27m11.2999386s) [noise=true , address=145.79.227.29:4942 , piece_id=QOOXF6G2OGBNJLA64B2YNVVJKAZSP6HM5ZIEI4ZFJKWZUSQAR3XA, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.828821058 (24h27m11.300031807s) [noise=true , address=145.79.227.16:4910 , piece_id=36JF4CMQH5JO66BY5A53QNZQ6RIFTHZ6MD2QLBGHLHML5VBFGK7A, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.829248923 (24h27m11.300459652s) [noise=true , address=145.79.227.43:4157 , piece_id=VWS32XVNWH4R5PZDDV4FGP5ONZQZNLMU5YYB7QYS43GDJSDFX2OQ, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.829618296 (24h27m11.300829035s) [noise=true , address=145.79.227.38:4978 , piece_id=JLYEPL2WLUJIIUHI3FSOWIT5QG3ZQG3NNOZZ3TNMAOEWLZJNN6AA, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.830015262 (24h27m11.301226001s) [noise=true , address=77.90.31.19:4843 , piece_id=36QLYXSVN6L2FDKVX4WS7MRTK54XLB2JUTW2NKHCE7E6KZGTLUKQ, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.830278002 (24h27m11.301488741s) [noise=true , address=145.79.227.21:4475 , piece_id=YZRHRIA7WWLZKDX3BZ2FSUOKJ3MX7D7Q2GB6CHKVQKGXPQCUNC6A, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.830415063 (24h27m11.301625802s) [noise=true , address=145.79.227.8:4329 , piece_id=FBD34B6US36TX6YHN5UEBKOWIVOOP4G7B6SSYKN6Q3J5YNE7OBHA, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.830692902 (24h27m11.301903641s) [noise=true , address=145.79.227.9:4366 , piece_id=GDCC3W66FFB27QD5MRCG2MBJXZBWXRIWG7A4L3XMFTIRNN2NYPUQ, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.830811337 (24h27m11.302022076s) [noise=true , address=77.90.31.10:4520 , piece_id=DGMWUPIGVLYJ4IH2CXCHA3U7FHDJF7APG73XN6TUEWRBB3WZOURQ, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.831349433 (24h27m11.302560172s) [noise=true , address=77.90.31.18:4807 , piece_id=WJ3AOAEE3IG3T3QQRFSKTOJCYVST4263Y5UN6XVOVTSO5ALOVTCQ, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.831805731 (24h27m11.30301647s) [noise=true , address=145.79.227.43:4158 , piece_id=74VKW6UWNPLSDKXJ6AG6TTUSDBCW4K4BAGMFIDNIV55AIWRBVB2Q, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.832172379 (24h27m11.303383118s) [noise=true , address=145.79.227.39:4014 , piece_id=ZLJ7L4OXXXNMJP2NTDU5HNE7NDCFL6JNWLXB6KVZHHFZXF5AT4CA, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.833230595 (24h27m11.304441334s) [noise=true , address=145.79.227.34:4905 , piece_id=ULLE3QB2QSCSYQWZ4OSBDJHC3X7AOHJGRWRA4DUXAMOYBDJRYWRQ, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.833972006 (24h27m11.305182755s) [noise=true , address=145.79.227.10:4401 , piece_id=TLP7B6MNA7W35UJQKW5SW6F43YXXMUFXOUKW4DKSOWOPIKLSSVUA, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:35.930150729 (24h27m12.401361488s) [noise=true , address=77.90.31.19:4843 , piece_id=36QLYXSVN6L2FDKVX4WS7MRTK54XLB2JUTW2NKHCE7E6KZGTLUKQ, seg_pos={0 0}, upload=21507]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: storage node overloaded, request limit: 2000 Dec 4 17:22:36.479057771 (24h27m12.9502685s) [upload=21507]: Next returned nil segment Dec 4 17:22:36.479122765 (24h27m12.950333494s) [upload=21507]: Waiting for error group managing segments... Dec 4 17:22:36.876009453 (24h27m13.347220192s) [noise=true , address=77.90.31.21:4878 , piece_id=GHU3SZUB5WAMW37CYCUFJ6HOPMXRNZGIJYDC4DJ3WOHV7YYWCVEA, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:36.891015988 (24h27m13.362226717s) [noise=true , address=77.90.31.14:4661 , piece_id=D5L5WWE5DGEYKP5M2NM74EGSX5BLEP4DX5OBM2Z4Q2QTCSGA4WXQ, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:36.943731919 (24h27m13.414942658s) [noise=true , address=77.90.31.10:4520 , piece_id=DGMWUPIGVLYJ4IH2CXCHA3U7FHDJF7APG73XN6TUEWRBB3WZOURQ, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:36.981669560 (24h27m13.452880289s) [noise=true , address=145.79.227.16:4910 , piece_id=36JF4CMQH5JO66BY5A53QNZQ6RIFTHZ6MD2QLBGHLHML5VBFGK7A, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:36.981785620 (24h27m13.452996359s) [noise=true , address=145.79.227.21:4475 , piece_id=YZRHRIA7WWLZKDX3BZ2FSUOKJ3MX7D7Q2GB6CHKVQKGXPQCUNC6A, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:36.984955717 (24h27m13.456166456s) [noise=true , address=77.90.31.18:4807 , piece_id=WJ3AOAEE3IG3T3QQRFSKTOJCYVST4263Y5UN6XVOVTSO5ALOVTCQ, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:36.994452571 (24h27m13.4656633s) [noise=true , address=145.79.227.34:4905 , piece_id=ULLE3QB2QSCSYQWZ4OSBDJHC3X7AOHJGRWRA4DUXAMOYBDJRYWRQ, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:37.011055396 (24h27m13.482266125s) [noise=true , address=145.79.227.38:4978 , piece_id=JLYEPL2WLUJIIUHI3FSOWIT5QG3ZQG3NNOZZ3TNMAOEWLZJNN6AA, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:37.021499513 (24h27m13.492710252s) [noise=true , address=145.79.227.8:4329 , piece_id=FBD34B6US36TX6YHN5UEBKOWIVOOP4G7B6SSYKN6Q3J5YNE7OBHA, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:37.043392031 (24h27m13.51460277s) [noise=true , address=145.79.227.39:4014 , piece_id=ZLJ7L4OXXXNMJP2NTDU5HNE7NDCFL6JNWLXB6KVZHHFZXF5AT4CA, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:37.047232213 (24h27m13.518442952s) [noise=true , address=145.79.227.29:4942 , piece_id=QOOXF6G2OGBNJLA64B2YNVVJKAZSP6HM5ZIEI4ZFJKWZUSQAR3XA, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:37.315831267 (24h27m13.787042006s) [noise=true , address=145.79.227.10:4401 , piece_id=TLP7B6MNA7W35UJQKW5SW6F43YXXMUFXOUKW4DKSOWOPIKLSSVUA, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:37.332403453 (24h27m13.803614192s) [noise=true , address=145.79.227.43:4157 , piece_id=VWS32XVNWH4R5PZDDV4FGP5ONZQZNLMU5YYB7QYS43GDJSDFX2OQ, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:37.332432619 (24h27m13.803643358s) [seg_pos={0 0}, upload=21507]: Segment reached optimal threshold of 13 pieces. Dec 4 17:22:37.332650384 (24h27m13.803861123s) [noise=true , address=145.79.227.9:4366 , piece_id=GDCC3W66FFB27QD5MRCG2MBJXZBWXRIWG7A4L3XMFTIRNN2NYPUQ, seg_pos={0 0}, upload=21507]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy): protocol: context canceled Dec 4 17:22:37.332660022 (24h27m13.803870751s) [noise=true , address=145.79.227.9:4366 , piece_id=GDCC3W66FFB27QD5MRCG2MBJXZBWXRIWG7A4L3XMFTIRNN2NYPUQ, seg_pos={0 0}, upload=21507]: Long tail cancellation due to optimal threshold Dec 4 17:22:37.332942179 (24h27m13.804152938s) [noise=true , address=145.79.227.43:4158 , piece_id=74VKW6UWNPLSDKXJ6AG6TTUSDBCW4K4BAGMFIDNIV55AIWRBVB2Q, seg_pos={0 0}, upload=21507]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vJZrSvPvwebw9V2tTbvkPEmLeLjryBUnq9TL6NwdPsHU9xcq2): protocol: expected piece hash; context canceled; EOF Dec 4 17:22:37.332954993 (24h27m13.804165732s) [noise=true , address=145.79.227.43:4158 , piece_id=74VKW6UWNPLSDKXJ6AG6TTUSDBCW4K4BAGMFIDNIV55AIWRBVB2Q, seg_pos={0 0}, upload=21507]: Long tail cancellation due to optimal threshold Dec 4 17:22:37.332976635 (24h27m13.804187384s) [seg_pos={0 0}, upload=21507]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 4 17:22:37.333114427 (24h27m13.804325166s) [upload=21507]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 4 17:22:37.333121821 (24h27m13.80433255s) [upload=21507]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 4 17:22:37.958438444 (24h27m14.429649173s) [upload=21507]: Done uploading segments... Dec 4 17:22:37.958456208 (24h27m14.429666937s) [upload=21507]: Upload finished. err: <nil> Duration: 5.733366333s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 5.653968356s trace: https://jaeger.dev.storj.io/trace/6d15aafc5ce7b199 FAIL (2025-12-04 17:40:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, Piece ID: FBD34B6US36TX6YHN5UEBKOWIVOOP4G7B6SSYKN6Q3J5YNE7OBHA) context deadline exceeded; piecestore: (Node ID: 12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9, Piece ID: ULLE3QB2QSCSYQWZ4OSBDJHC3X7AOHJGRWRA4DUXAMOYBDJRYWRQ) context deadline exceeded; piecestore: (Node ID: 1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, Piece ID: TLP7B6MNA7W35UJQKW5SW6F43YXXMUFXOUKW4DKSOWOPIKLSSVUA) context deadline exceeded; piecestore: (Node ID: 17o881pUMuG7K2dv83k7RkrJCrmM7kgUiWesgam4GEw21KVUn7, Piece ID: QOOXF6G2OGBNJLA64B2YNVVJKAZSP6HM5ZIEI4ZFJKWZUSQAR3XA) context deadline exceeded; piecestore: (Node ID: 1ZrzADk2mKG4R9tjxUKjQeJvb4WmfjLksGFyvstHRfYbBC7XNz, Piece ID: VWS32XVNWH4R5PZDDV4FGP5ONZQZNLMU5YYB7QYS43GDJSDFX2OQ) context deadline exceeded ======LOGS====== Dec 4 17:22:34.320746327 (24h27m10.791957066s) [upload=21507]: Starting upload Dec 4 17:22:34.320860715 (24h27m10.792071454s) [upload=21507]: Done starting upload Dec 4 17:22:34.321208968 (24h27m10.792419707s) [upload=21507]: Uploading segments... Dec 4 17:22:34.321325259 (24h27m10.792536018s) [upload=21507]: Got next segment. Inline: false Dec 4 17:22:34.321394982 (24h27m10.792605711s) [upload=21507]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 17:22:34.321401375 (24h27m10.792612114s) [upload=21507]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 17:22:34.827776318 (24h27m11.298987057s) [seg_pos={0 0}, upload=21507]: Begin upload segment... Dec 4 17:22:34.827838226 (24h27m11.299048966s) [seg_pos={0 0}, upload=21507]: Done begin upload segment. Dec 4 17:22:34.827877361 (24h27m11.29908811s) [noise=true , address=77.90.31.14:4661 , piece_id=D5L5WWE5DGEYKP5M2NM74EGSX5BLEP4DX5OBM2Z4Q2QTCSGA4WXQ, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.827887630 (24h27m11.299098359s) [noise=true , address=77.90.31.21:4878 , piece_id=GHU3SZUB5WAMW37CYCUFJ6HOPMXRNZGIJYDC4DJ3WOHV7YYWCVEA, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.828727861 (24h27m11.2999386s) [noise=true , address=145.79.227.29:4942 , piece_id=QOOXF6G2OGBNJLA64B2YNVVJKAZSP6HM5ZIEI4ZFJKWZUSQAR3XA, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.828821058 (24h27m11.300031807s) [noise=true , address=145.79.227.16:4910 , piece_id=36JF4CMQH5JO66BY5A53QNZQ6RIFTHZ6MD2QLBGHLHML5VBFGK7A, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.829248923 (24h27m11.300459652s) [noise=true , address=145.79.227.43:4157 , piece_id=VWS32XVNWH4R5PZDDV4FGP5ONZQZNLMU5YYB7QYS43GDJSDFX2OQ, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.829618296 (24h27m11.300829035s) [noise=true , address=145.79.227.38:4978 , piece_id=JLYEPL2WLUJIIUHI3FSOWIT5QG3ZQG3NNOZZ3TNMAOEWLZJNN6AA, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.830015262 (24h27m11.301226001s) [noise=true , address=77.90.31.19:4843 , piece_id=36QLYXSVN6L2FDKVX4WS7MRTK54XLB2JUTW2NKHCE7E6KZGTLUKQ, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.830278002 (24h27m11.301488741s) [noise=true , address=145.79.227.21:4475 , piece_id=YZRHRIA7WWLZKDX3BZ2FSUOKJ3MX7D7Q2GB6CHKVQKGXPQCUNC6A, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.830415063 (24h27m11.301625802s) [noise=true , address=145.79.227.8:4329 , piece_id=FBD34B6US36TX6YHN5UEBKOWIVOOP4G7B6SSYKN6Q3J5YNE7OBHA, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.830692902 (24h27m11.301903641s) [noise=true , address=145.79.227.9:4366 , piece_id=GDCC3W66FFB27QD5MRCG2MBJXZBWXRIWG7A4L3XMFTIRNN2NYPUQ, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.830811337 (24h27m11.302022076s) [noise=true , address=77.90.31.10:4520 , piece_id=DGMWUPIGVLYJ4IH2CXCHA3U7FHDJF7APG73XN6TUEWRBB3WZOURQ, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.831349433 (24h27m11.302560172s) [noise=true , address=77.90.31.18:4807 , piece_id=WJ3AOAEE3IG3T3QQRFSKTOJCYVST4263Y5UN6XVOVTSO5ALOVTCQ, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.831805731 (24h27m11.30301647s) [noise=true , address=145.79.227.43:4158 , piece_id=74VKW6UWNPLSDKXJ6AG6TTUSDBCW4K4BAGMFIDNIV55AIWRBVB2Q, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.832172379 (24h27m11.303383118s) [noise=true , address=145.79.227.39:4014 , piece_id=ZLJ7L4OXXXNMJP2NTDU5HNE7NDCFL6JNWLXB6KVZHHFZXF5AT4CA, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.833230595 (24h27m11.304441334s) [noise=true , address=145.79.227.34:4905 , piece_id=ULLE3QB2QSCSYQWZ4OSBDJHC3X7AOHJGRWRA4DUXAMOYBDJRYWRQ, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.833972006 (24h27m11.305182755s) [noise=true , address=145.79.227.10:4401 , piece_id=TLP7B6MNA7W35UJQKW5SW6F43YXXMUFXOUKW4DKSOWOPIKLSSVUA, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:35.930150729 (24h27m12.401361488s) [noise=true , address=77.90.31.19:4843 , piece_id=36QLYXSVN6L2FDKVX4WS7MRTK54XLB2JUTW2NKHCE7E6KZGTLUKQ, seg_pos={0 0}, upload=21507]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: storage node overloaded, request limit: 2000 Dec 4 17:22:36.479057771 (24h27m12.9502685s) [upload=21507]: Next returned nil segment Dec 4 17:22:36.479122765 (24h27m12.950333494s) [upload=21507]: Waiting for error group managing segments... Dec 4 17:22:36.876009453 (24h27m13.347220192s) [noise=true , address=77.90.31.21:4878 , piece_id=GHU3SZUB5WAMW37CYCUFJ6HOPMXRNZGIJYDC4DJ3WOHV7YYWCVEA, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:36.891015988 (24h27m13.362226717s) [noise=true , address=77.90.31.14:4661 , piece_id=D5L5WWE5DGEYKP5M2NM74EGSX5BLEP4DX5OBM2Z4Q2QTCSGA4WXQ, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:36.943731919 (24h27m13.414942658s) [noise=true , address=77.90.31.10:4520 , piece_id=DGMWUPIGVLYJ4IH2CXCHA3U7FHDJF7APG73XN6TUEWRBB3WZOURQ, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:36.981669560 (24h27m13.452880289s) [noise=true , address=145.79.227.16:4910 , piece_id=36JF4CMQH5JO66BY5A53QNZQ6RIFTHZ6MD2QLBGHLHML5VBFGK7A, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:36.981785620 (24h27m13.452996359s) [noise=true , address=145.79.227.21:4475 , piece_id=YZRHRIA7WWLZKDX3BZ2FSUOKJ3MX7D7Q2GB6CHKVQKGXPQCUNC6A, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:36.984955717 (24h27m13.456166456s) [noise=true , address=77.90.31.18:4807 , piece_id=WJ3AOAEE3IG3T3QQRFSKTOJCYVST4263Y5UN6XVOVTSO5ALOVTCQ, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:36.994452571 (24h27m13.4656633s) [noise=true , address=145.79.227.34:4905 , piece_id=ULLE3QB2QSCSYQWZ4OSBDJHC3X7AOHJGRWRA4DUXAMOYBDJRYWRQ, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:37.011055396 (24h27m13.482266125s) [noise=true , address=145.79.227.38:4978 , piece_id=JLYEPL2WLUJIIUHI3FSOWIT5QG3ZQG3NNOZZ3TNMAOEWLZJNN6AA, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:37.021499513 (24h27m13.492710252s) [noise=true , address=145.79.227.8:4329 , piece_id=FBD34B6US36TX6YHN5UEBKOWIVOOP4G7B6SSYKN6Q3J5YNE7OBHA, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:37.043392031 (24h27m13.51460277s) [noise=true , address=145.79.227.39:4014 , piece_id=ZLJ7L4OXXXNMJP2NTDU5HNE7NDCFL6JNWLXB6KVZHHFZXF5AT4CA, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:37.047232213 (24h27m13.518442952s) [noise=true , address=145.79.227.29:4942 , piece_id=QOOXF6G2OGBNJLA64B2YNVVJKAZSP6HM5ZIEI4ZFJKWZUSQAR3XA, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:37.315831267 (24h27m13.787042006s) [noise=true , address=145.79.227.10:4401 , piece_id=TLP7B6MNA7W35UJQKW5SW6F43YXXMUFXOUKW4DKSOWOPIKLSSVUA, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:37.332403453 (24h27m13.803614192s) [noise=true , address=145.79.227.43:4157 , piece_id=VWS32XVNWH4R5PZDDV4FGP5ONZQZNLMU5YYB7QYS43GDJSDFX2OQ, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:37.332432619 (24h27m13.803643358s) [seg_pos={0 0}, upload=21507]: Segment reached optimal threshold of 13 pieces. Dec 4 17:22:37.332650384 (24h27m13.803861123s) [noise=true , address=145.79.227.9:4366 , piece_id=GDCC3W66FFB27QD5MRCG2MBJXZBWXRIWG7A4L3XMFTIRNN2NYPUQ, seg_pos={0 0}, upload=21507]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy): protocol: context canceled Dec 4 17:22:37.332660022 (24h27m13.803870751s) [noise=true , address=145.79.227.9:4366 , piece_id=GDCC3W66FFB27QD5MRCG2MBJXZBWXRIWG7A4L3XMFTIRNN2NYPUQ, seg_pos={0 0}, upload=21507]: Long tail cancellation due to optimal threshold Dec 4 17:22:37.332942179 (24h27m13.804152938s) [noise=true , address=145.79.227.43:4158 , piece_id=74VKW6UWNPLSDKXJ6AG6TTUSDBCW4K4BAGMFIDNIV55AIWRBVB2Q, seg_pos={0 0}, upload=21507]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vJZrSvPvwebw9V2tTbvkPEmLeLjryBUnq9TL6NwdPsHU9xcq2): protocol: expected piece hash; context canceled; EOF Dec 4 17:22:37.332954993 (24h27m13.804165732s) [noise=true , address=145.79.227.43:4158 , piece_id=74VKW6UWNPLSDKXJ6AG6TTUSDBCW4K4BAGMFIDNIV55AIWRBVB2Q, seg_pos={0 0}, upload=21507]: Long tail cancellation due to optimal threshold Dec 4 17:22:37.332976635 (24h27m13.804187384s) [seg_pos={0 0}, upload=21507]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 4 17:22:37.333114427 (24h27m13.804325166s) [upload=21507]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 4 17:22:37.333121821 (24h27m13.80433255s) [upload=21507]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 4 17:22:37.958438444 (24h27m14.429649173s) [upload=21507]: Done uploading segments... Dec 4 17:22:37.958456208 (24h27m14.429666937s) [upload=21507]: Upload finished. err: <nil> Duration: 5.653998463s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 5.726166535s trace: https://jaeger.dev.storj.io/trace/6d15aafc5ce7b199 FAIL (2025-12-04 17:40:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, Piece ID: FBD34B6US36TX6YHN5UEBKOWIVOOP4G7B6SSYKN6Q3J5YNE7OBHA) context deadline exceeded; piecestore: (Node ID: 12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9, Piece ID: ULLE3QB2QSCSYQWZ4OSBDJHC3X7AOHJGRWRA4DUXAMOYBDJRYWRQ) context deadline exceeded; piecestore: (Node ID: 1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, Piece ID: TLP7B6MNA7W35UJQKW5SW6F43YXXMUFXOUKW4DKSOWOPIKLSSVUA) context deadline exceeded; piecestore: (Node ID: 17o881pUMuG7K2dv83k7RkrJCrmM7kgUiWesgam4GEw21KVUn7, Piece ID: QOOXF6G2OGBNJLA64B2YNVVJKAZSP6HM5ZIEI4ZFJKWZUSQAR3XA) context deadline exceeded; piecestore: (Node ID: 1ZrzADk2mKG4R9tjxUKjQeJvb4WmfjLksGFyvstHRfYbBC7XNz, Piece ID: VWS32XVNWH4R5PZDDV4FGP5ONZQZNLMU5YYB7QYS43GDJSDFX2OQ) context deadline exceeded ======LOGS====== Dec 4 17:22:34.320746327 (24h27m10.791957066s) [upload=21507]: Starting upload Dec 4 17:22:34.320860715 (24h27m10.792071454s) [upload=21507]: Done starting upload Dec 4 17:22:34.321208968 (24h27m10.792419707s) [upload=21507]: Uploading segments... Dec 4 17:22:34.321325259 (24h27m10.792536018s) [upload=21507]: Got next segment. Inline: false Dec 4 17:22:34.321394982 (24h27m10.792605711s) [upload=21507]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 17:22:34.321401375 (24h27m10.792612114s) [upload=21507]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 17:22:34.827776318 (24h27m11.298987057s) [seg_pos={0 0}, upload=21507]: Begin upload segment... Dec 4 17:22:34.827838226 (24h27m11.299048966s) [seg_pos={0 0}, upload=21507]: Done begin upload segment. Dec 4 17:22:34.827877361 (24h27m11.29908811s) [noise=true , address=77.90.31.14:4661 , piece_id=D5L5WWE5DGEYKP5M2NM74EGSX5BLEP4DX5OBM2Z4Q2QTCSGA4WXQ, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.827887630 (24h27m11.299098359s) [noise=true , address=77.90.31.21:4878 , piece_id=GHU3SZUB5WAMW37CYCUFJ6HOPMXRNZGIJYDC4DJ3WOHV7YYWCVEA, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.828727861 (24h27m11.2999386s) [noise=true , address=145.79.227.29:4942 , piece_id=QOOXF6G2OGBNJLA64B2YNVVJKAZSP6HM5ZIEI4ZFJKWZUSQAR3XA, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.828821058 (24h27m11.300031807s) [noise=true , address=145.79.227.16:4910 , piece_id=36JF4CMQH5JO66BY5A53QNZQ6RIFTHZ6MD2QLBGHLHML5VBFGK7A, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.829248923 (24h27m11.300459652s) [noise=true , address=145.79.227.43:4157 , piece_id=VWS32XVNWH4R5PZDDV4FGP5ONZQZNLMU5YYB7QYS43GDJSDFX2OQ, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.829618296 (24h27m11.300829035s) [noise=true , address=145.79.227.38:4978 , piece_id=JLYEPL2WLUJIIUHI3FSOWIT5QG3ZQG3NNOZZ3TNMAOEWLZJNN6AA, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.830015262 (24h27m11.301226001s) [noise=true , address=77.90.31.19:4843 , piece_id=36QLYXSVN6L2FDKVX4WS7MRTK54XLB2JUTW2NKHCE7E6KZGTLUKQ, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.830278002 (24h27m11.301488741s) [noise=true , address=145.79.227.21:4475 , piece_id=YZRHRIA7WWLZKDX3BZ2FSUOKJ3MX7D7Q2GB6CHKVQKGXPQCUNC6A, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.830415063 (24h27m11.301625802s) [noise=true , address=145.79.227.8:4329 , piece_id=FBD34B6US36TX6YHN5UEBKOWIVOOP4G7B6SSYKN6Q3J5YNE7OBHA, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.830692902 (24h27m11.301903641s) [noise=true , address=145.79.227.9:4366 , piece_id=GDCC3W66FFB27QD5MRCG2MBJXZBWXRIWG7A4L3XMFTIRNN2NYPUQ, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.830811337 (24h27m11.302022076s) [noise=true , address=77.90.31.10:4520 , piece_id=DGMWUPIGVLYJ4IH2CXCHA3U7FHDJF7APG73XN6TUEWRBB3WZOURQ, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.831349433 (24h27m11.302560172s) [noise=true , address=77.90.31.18:4807 , piece_id=WJ3AOAEE3IG3T3QQRFSKTOJCYVST4263Y5UN6XVOVTSO5ALOVTCQ, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.831805731 (24h27m11.30301647s) [noise=true , address=145.79.227.43:4158 , piece_id=74VKW6UWNPLSDKXJ6AG6TTUSDBCW4K4BAGMFIDNIV55AIWRBVB2Q, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.832172379 (24h27m11.303383118s) [noise=true , address=145.79.227.39:4014 , piece_id=ZLJ7L4OXXXNMJP2NTDU5HNE7NDCFL6JNWLXB6KVZHHFZXF5AT4CA, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.833230595 (24h27m11.304441334s) [noise=true , address=145.79.227.34:4905 , piece_id=ULLE3QB2QSCSYQWZ4OSBDJHC3X7AOHJGRWRA4DUXAMOYBDJRYWRQ, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.833972006 (24h27m11.305182755s) [noise=true , address=145.79.227.10:4401 , piece_id=TLP7B6MNA7W35UJQKW5SW6F43YXXMUFXOUKW4DKSOWOPIKLSSVUA, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:35.930150729 (24h27m12.401361488s) [noise=true , address=77.90.31.19:4843 , piece_id=36QLYXSVN6L2FDKVX4WS7MRTK54XLB2JUTW2NKHCE7E6KZGTLUKQ, seg_pos={0 0}, upload=21507]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: storage node overloaded, request limit: 2000 Dec 4 17:22:36.479057771 (24h27m12.9502685s) [upload=21507]: Next returned nil segment Dec 4 17:22:36.479122765 (24h27m12.950333494s) [upload=21507]: Waiting for error group managing segments... Dec 4 17:22:36.876009453 (24h27m13.347220192s) [noise=true , address=77.90.31.21:4878 , piece_id=GHU3SZUB5WAMW37CYCUFJ6HOPMXRNZGIJYDC4DJ3WOHV7YYWCVEA, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:36.891015988 (24h27m13.362226717s) [noise=true , address=77.90.31.14:4661 , piece_id=D5L5WWE5DGEYKP5M2NM74EGSX5BLEP4DX5OBM2Z4Q2QTCSGA4WXQ, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:36.943731919 (24h27m13.414942658s) [noise=true , address=77.90.31.10:4520 , piece_id=DGMWUPIGVLYJ4IH2CXCHA3U7FHDJF7APG73XN6TUEWRBB3WZOURQ, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:36.981669560 (24h27m13.452880289s) [noise=true , address=145.79.227.16:4910 , piece_id=36JF4CMQH5JO66BY5A53QNZQ6RIFTHZ6MD2QLBGHLHML5VBFGK7A, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:36.981785620 (24h27m13.452996359s) [noise=true , address=145.79.227.21:4475 , piece_id=YZRHRIA7WWLZKDX3BZ2FSUOKJ3MX7D7Q2GB6CHKVQKGXPQCUNC6A, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:36.984955717 (24h27m13.456166456s) [noise=true , address=77.90.31.18:4807 , piece_id=WJ3AOAEE3IG3T3QQRFSKTOJCYVST4263Y5UN6XVOVTSO5ALOVTCQ, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:36.994452571 (24h27m13.4656633s) [noise=true , address=145.79.227.34:4905 , piece_id=ULLE3QB2QSCSYQWZ4OSBDJHC3X7AOHJGRWRA4DUXAMOYBDJRYWRQ, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:37.011055396 (24h27m13.482266125s) [noise=true , address=145.79.227.38:4978 , piece_id=JLYEPL2WLUJIIUHI3FSOWIT5QG3ZQG3NNOZZ3TNMAOEWLZJNN6AA, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:37.021499513 (24h27m13.492710252s) [noise=true , address=145.79.227.8:4329 , piece_id=FBD34B6US36TX6YHN5UEBKOWIVOOP4G7B6SSYKN6Q3J5YNE7OBHA, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:37.043392031 (24h27m13.51460277s) [noise=true , address=145.79.227.39:4014 , piece_id=ZLJ7L4OXXXNMJP2NTDU5HNE7NDCFL6JNWLXB6KVZHHFZXF5AT4CA, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:37.047232213 (24h27m13.518442952s) [noise=true , address=145.79.227.29:4942 , piece_id=QOOXF6G2OGBNJLA64B2YNVVJKAZSP6HM5ZIEI4ZFJKWZUSQAR3XA, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:37.315831267 (24h27m13.787042006s) [noise=true , address=145.79.227.10:4401 , piece_id=TLP7B6MNA7W35UJQKW5SW6F43YXXMUFXOUKW4DKSOWOPIKLSSVUA, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:37.332403453 (24h27m13.803614192s) [noise=true , address=145.79.227.43:4157 , piece_id=VWS32XVNWH4R5PZDDV4FGP5ONZQZNLMU5YYB7QYS43GDJSDFX2OQ, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:37.332432619 (24h27m13.803643358s) [seg_pos={0 0}, upload=21507]: Segment reached optimal threshold of 13 pieces. Dec 4 17:22:37.332650384 (24h27m13.803861123s) [noise=true , address=145.79.227.9:4366 , piece_id=GDCC3W66FFB27QD5MRCG2MBJXZBWXRIWG7A4L3XMFTIRNN2NYPUQ, seg_pos={0 0}, upload=21507]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy): protocol: context canceled Dec 4 17:22:37.332660022 (24h27m13.803870751s) [noise=true , address=145.79.227.9:4366 , piece_id=GDCC3W66FFB27QD5MRCG2MBJXZBWXRIWG7A4L3XMFTIRNN2NYPUQ, seg_pos={0 0}, upload=21507]: Long tail cancellation due to optimal threshold Dec 4 17:22:37.332942179 (24h27m13.804152938s) [noise=true , address=145.79.227.43:4158 , piece_id=74VKW6UWNPLSDKXJ6AG6TTUSDBCW4K4BAGMFIDNIV55AIWRBVB2Q, seg_pos={0 0}, upload=21507]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vJZrSvPvwebw9V2tTbvkPEmLeLjryBUnq9TL6NwdPsHU9xcq2): protocol: expected piece hash; context canceled; EOF Dec 4 17:22:37.332954993 (24h27m13.804165732s) [noise=true , address=145.79.227.43:4158 , piece_id=74VKW6UWNPLSDKXJ6AG6TTUSDBCW4K4BAGMFIDNIV55AIWRBVB2Q, seg_pos={0 0}, upload=21507]: Long tail cancellation due to optimal threshold Dec 4 17:22:37.332976635 (24h27m13.804187384s) [seg_pos={0 0}, upload=21507]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 4 17:22:37.333114427 (24h27m13.804325166s) [upload=21507]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 4 17:22:37.333121821 (24h27m13.80433255s) [upload=21507]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 4 17:22:37.958438444 (24h27m14.429649173s) [upload=21507]: Done uploading segments... Dec 4 17:22:37.958456208 (24h27m14.429666937s) [upload=21507]: Upload finished. err: <nil> Duration: 5.726189568s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.........X..................................................] duration: 3.388841866s trace: https://jaeger.dev.storj.io/trace/1f3e769edbb82b7a FAIL (2025-12-04 17:39:58 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.388863196s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 5.681146147s trace: https://jaeger.dev.storj.io/trace/400044794bd63bf0 FAIL (2025-12-04 17:39:47 UTC) orbiter warning: read tcp 188.166.225.170:54202->45.38.24.2:443: read: connection reset by peer Duration: 5.681176366s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 5.762200447s trace: https://jaeger.dev.storj.io/trace/400044794bd63bf0 FAIL (2025-12-04 17:39:17 UTC) orbiter warning: read tcp 188.166.225.170:54202->45.38.24.2:443: read: connection reset by peer Duration: 5.762246325s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..........X.................................................] duration: 3.460495744s trace: https://jaeger.dev.storj.io/trace/1f3e769edbb82b7a FAIL (2025-12-04 17:39:13 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.46052495s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...........X................................................] duration: 3.362892922s trace: https://jaeger.dev.storj.io/trace/1f3e769edbb82b7a FAIL (2025-12-04 17:38:53 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.362921666s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 5.937125168s trace: https://jaeger.dev.storj.io/trace/6d15aafc5ce7b199 FAIL (2025-12-04 17:38:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, Piece ID: FBD34B6US36TX6YHN5UEBKOWIVOOP4G7B6SSYKN6Q3J5YNE7OBHA) context deadline exceeded; piecestore: (Node ID: 12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9, Piece ID: ULLE3QB2QSCSYQWZ4OSBDJHC3X7AOHJGRWRA4DUXAMOYBDJRYWRQ) context deadline exceeded; piecestore: (Node ID: 1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, Piece ID: TLP7B6MNA7W35UJQKW5SW6F43YXXMUFXOUKW4DKSOWOPIKLSSVUA) context deadline exceeded; piecestore: (Node ID: 17o881pUMuG7K2dv83k7RkrJCrmM7kgUiWesgam4GEw21KVUn7, Piece ID: QOOXF6G2OGBNJLA64B2YNVVJKAZSP6HM5ZIEI4ZFJKWZUSQAR3XA) context deadline exceeded; piecestore: (Node ID: 1ZrzADk2mKG4R9tjxUKjQeJvb4WmfjLksGFyvstHRfYbBC7XNz, Piece ID: VWS32XVNWH4R5PZDDV4FGP5ONZQZNLMU5YYB7QYS43GDJSDFX2OQ) context deadline exceeded ======LOGS====== Dec 4 17:22:34.320746327 (24h27m10.791957066s) [upload=21507]: Starting upload Dec 4 17:22:34.320860715 (24h27m10.792071454s) [upload=21507]: Done starting upload Dec 4 17:22:34.321208968 (24h27m10.792419707s) [upload=21507]: Uploading segments... Dec 4 17:22:34.321325259 (24h27m10.792536018s) [upload=21507]: Got next segment. Inline: false Dec 4 17:22:34.321394982 (24h27m10.792605711s) [upload=21507]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 17:22:34.321401375 (24h27m10.792612114s) [upload=21507]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 17:22:34.827776318 (24h27m11.298987057s) [seg_pos={0 0}, upload=21507]: Begin upload segment... Dec 4 17:22:34.827838226 (24h27m11.299048966s) [seg_pos={0 0}, upload=21507]: Done begin upload segment. Dec 4 17:22:34.827877361 (24h27m11.29908811s) [noise=true , address=77.90.31.14:4661 , piece_id=D5L5WWE5DGEYKP5M2NM74EGSX5BLEP4DX5OBM2Z4Q2QTCSGA4WXQ, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.827887630 (24h27m11.299098359s) [noise=true , address=77.90.31.21:4878 , piece_id=GHU3SZUB5WAMW37CYCUFJ6HOPMXRNZGIJYDC4DJ3WOHV7YYWCVEA, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.828727861 (24h27m11.2999386s) [noise=true , address=145.79.227.29:4942 , piece_id=QOOXF6G2OGBNJLA64B2YNVVJKAZSP6HM5ZIEI4ZFJKWZUSQAR3XA, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.828821058 (24h27m11.300031807s) [noise=true , address=145.79.227.16:4910 , piece_id=36JF4CMQH5JO66BY5A53QNZQ6RIFTHZ6MD2QLBGHLHML5VBFGK7A, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.829248923 (24h27m11.300459652s) [noise=true , address=145.79.227.43:4157 , piece_id=VWS32XVNWH4R5PZDDV4FGP5ONZQZNLMU5YYB7QYS43GDJSDFX2OQ, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.829618296 (24h27m11.300829035s) [noise=true , address=145.79.227.38:4978 , piece_id=JLYEPL2WLUJIIUHI3FSOWIT5QG3ZQG3NNOZZ3TNMAOEWLZJNN6AA, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.830015262 (24h27m11.301226001s) [noise=true , address=77.90.31.19:4843 , piece_id=36QLYXSVN6L2FDKVX4WS7MRTK54XLB2JUTW2NKHCE7E6KZGTLUKQ, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.830278002 (24h27m11.301488741s) [noise=true , address=145.79.227.21:4475 , piece_id=YZRHRIA7WWLZKDX3BZ2FSUOKJ3MX7D7Q2GB6CHKVQKGXPQCUNC6A, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.830415063 (24h27m11.301625802s) [noise=true , address=145.79.227.8:4329 , piece_id=FBD34B6US36TX6YHN5UEBKOWIVOOP4G7B6SSYKN6Q3J5YNE7OBHA, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.830692902 (24h27m11.301903641s) [noise=true , address=145.79.227.9:4366 , piece_id=GDCC3W66FFB27QD5MRCG2MBJXZBWXRIWG7A4L3XMFTIRNN2NYPUQ, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.830811337 (24h27m11.302022076s) [noise=true , address=77.90.31.10:4520 , piece_id=DGMWUPIGVLYJ4IH2CXCHA3U7FHDJF7APG73XN6TUEWRBB3WZOURQ, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.831349433 (24h27m11.302560172s) [noise=true , address=77.90.31.18:4807 , piece_id=WJ3AOAEE3IG3T3QQRFSKTOJCYVST4263Y5UN6XVOVTSO5ALOVTCQ, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.831805731 (24h27m11.30301647s) [noise=true , address=145.79.227.43:4158 , piece_id=74VKW6UWNPLSDKXJ6AG6TTUSDBCW4K4BAGMFIDNIV55AIWRBVB2Q, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.832172379 (24h27m11.303383118s) [noise=true , address=145.79.227.39:4014 , piece_id=ZLJ7L4OXXXNMJP2NTDU5HNE7NDCFL6JNWLXB6KVZHHFZXF5AT4CA, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.833230595 (24h27m11.304441334s) [noise=true , address=145.79.227.34:4905 , piece_id=ULLE3QB2QSCSYQWZ4OSBDJHC3X7AOHJGRWRA4DUXAMOYBDJRYWRQ, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.833972006 (24h27m11.305182755s) [noise=true , address=145.79.227.10:4401 , piece_id=TLP7B6MNA7W35UJQKW5SW6F43YXXMUFXOUKW4DKSOWOPIKLSSVUA, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:35.930150729 (24h27m12.401361488s) [noise=true , address=77.90.31.19:4843 , piece_id=36QLYXSVN6L2FDKVX4WS7MRTK54XLB2JUTW2NKHCE7E6KZGTLUKQ, seg_pos={0 0}, upload=21507]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: storage node overloaded, request limit: 2000 Dec 4 17:22:36.479057771 (24h27m12.9502685s) [upload=21507]: Next returned nil segment Dec 4 17:22:36.479122765 (24h27m12.950333494s) [upload=21507]: Waiting for error group managing segments... Dec 4 17:22:36.876009453 (24h27m13.347220192s) [noise=true , address=77.90.31.21:4878 , piece_id=GHU3SZUB5WAMW37CYCUFJ6HOPMXRNZGIJYDC4DJ3WOHV7YYWCVEA, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:36.891015988 (24h27m13.362226717s) [noise=true , address=77.90.31.14:4661 , piece_id=D5L5WWE5DGEYKP5M2NM74EGSX5BLEP4DX5OBM2Z4Q2QTCSGA4WXQ, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:36.943731919 (24h27m13.414942658s) [noise=true , address=77.90.31.10:4520 , piece_id=DGMWUPIGVLYJ4IH2CXCHA3U7FHDJF7APG73XN6TUEWRBB3WZOURQ, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:36.981669560 (24h27m13.452880289s) [noise=true , address=145.79.227.16:4910 , piece_id=36JF4CMQH5JO66BY5A53QNZQ6RIFTHZ6MD2QLBGHLHML5VBFGK7A, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:36.981785620 (24h27m13.452996359s) [noise=true , address=145.79.227.21:4475 , piece_id=YZRHRIA7WWLZKDX3BZ2FSUOKJ3MX7D7Q2GB6CHKVQKGXPQCUNC6A, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:36.984955717 (24h27m13.456166456s) [noise=true , address=77.90.31.18:4807 , piece_id=WJ3AOAEE3IG3T3QQRFSKTOJCYVST4263Y5UN6XVOVTSO5ALOVTCQ, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:36.994452571 (24h27m13.4656633s) [noise=true , address=145.79.227.34:4905 , piece_id=ULLE3QB2QSCSYQWZ4OSBDJHC3X7AOHJGRWRA4DUXAMOYBDJRYWRQ, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:37.011055396 (24h27m13.482266125s) [noise=true , address=145.79.227.38:4978 , piece_id=JLYEPL2WLUJIIUHI3FSOWIT5QG3ZQG3NNOZZ3TNMAOEWLZJNN6AA, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:37.021499513 (24h27m13.492710252s) [noise=true , address=145.79.227.8:4329 , piece_id=FBD34B6US36TX6YHN5UEBKOWIVOOP4G7B6SSYKN6Q3J5YNE7OBHA, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:37.043392031 (24h27m13.51460277s) [noise=true , address=145.79.227.39:4014 , piece_id=ZLJ7L4OXXXNMJP2NTDU5HNE7NDCFL6JNWLXB6KVZHHFZXF5AT4CA, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:37.047232213 (24h27m13.518442952s) [noise=true , address=145.79.227.29:4942 , piece_id=QOOXF6G2OGBNJLA64B2YNVVJKAZSP6HM5ZIEI4ZFJKWZUSQAR3XA, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:37.315831267 (24h27m13.787042006s) [noise=true , address=145.79.227.10:4401 , piece_id=TLP7B6MNA7W35UJQKW5SW6F43YXXMUFXOUKW4DKSOWOPIKLSSVUA, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:37.332403453 (24h27m13.803614192s) [noise=true , address=145.79.227.43:4157 , piece_id=VWS32XVNWH4R5PZDDV4FGP5ONZQZNLMU5YYB7QYS43GDJSDFX2OQ, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:37.332432619 (24h27m13.803643358s) [seg_pos={0 0}, upload=21507]: Segment reached optimal threshold of 13 pieces. Dec 4 17:22:37.332650384 (24h27m13.803861123s) [noise=true , address=145.79.227.9:4366 , piece_id=GDCC3W66FFB27QD5MRCG2MBJXZBWXRIWG7A4L3XMFTIRNN2NYPUQ, seg_pos={0 0}, upload=21507]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy): protocol: context canceled Dec 4 17:22:37.332660022 (24h27m13.803870751s) [noise=true , address=145.79.227.9:4366 , piece_id=GDCC3W66FFB27QD5MRCG2MBJXZBWXRIWG7A4L3XMFTIRNN2NYPUQ, seg_pos={0 0}, upload=21507]: Long tail cancellation due to optimal threshold Dec 4 17:22:37.332942179 (24h27m13.804152938s) [noise=true , address=145.79.227.43:4158 , piece_id=74VKW6UWNPLSDKXJ6AG6TTUSDBCW4K4BAGMFIDNIV55AIWRBVB2Q, seg_pos={0 0}, upload=21507]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vJZrSvPvwebw9V2tTbvkPEmLeLjryBUnq9TL6NwdPsHU9xcq2): protocol: expected piece hash; context canceled; EOF Dec 4 17:22:37.332954993 (24h27m13.804165732s) [noise=true , address=145.79.227.43:4158 , piece_id=74VKW6UWNPLSDKXJ6AG6TTUSDBCW4K4BAGMFIDNIV55AIWRBVB2Q, seg_pos={0 0}, upload=21507]: Long tail cancellation due to optimal threshold Dec 4 17:22:37.332976635 (24h27m13.804187384s) [seg_pos={0 0}, upload=21507]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 4 17:22:37.333114427 (24h27m13.804325166s) [upload=21507]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 4 17:22:37.333121821 (24h27m13.80433255s) [upload=21507]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 4 17:22:37.958438444 (24h27m14.429649173s) [upload=21507]: Done uploading segments... Dec 4 17:22:37.958456208 (24h27m14.429666937s) [upload=21507]: Upload finished. err: <nil> Duration: 5.937158742s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 5.688557359s trace: https://jaeger.dev.storj.io/trace/400044794bd63bf0 FAIL (2025-12-04 17:38:20 UTC) orbiter warning: read tcp 188.166.225.170:54202->45.38.24.2:443: read: connection reset by peer Duration: 5.688588488s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [............X...............................................] duration: 3.596235468s trace: https://jaeger.dev.storj.io/trace/1f3e769edbb82b7a FAIL (2025-12-04 17:38:11 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.596264364s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 5.693954036s trace: https://jaeger.dev.storj.io/trace/6d15aafc5ce7b199 FAIL (2025-12-04 17:37:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, Piece ID: FBD34B6US36TX6YHN5UEBKOWIVOOP4G7B6SSYKN6Q3J5YNE7OBHA) context deadline exceeded; piecestore: (Node ID: 12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9, Piece ID: ULLE3QB2QSCSYQWZ4OSBDJHC3X7AOHJGRWRA4DUXAMOYBDJRYWRQ) context deadline exceeded; piecestore: (Node ID: 1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, Piece ID: TLP7B6MNA7W35UJQKW5SW6F43YXXMUFXOUKW4DKSOWOPIKLSSVUA) context deadline exceeded; piecestore: (Node ID: 17o881pUMuG7K2dv83k7RkrJCrmM7kgUiWesgam4GEw21KVUn7, Piece ID: QOOXF6G2OGBNJLA64B2YNVVJKAZSP6HM5ZIEI4ZFJKWZUSQAR3XA) context deadline exceeded; piecestore: (Node ID: 1ZrzADk2mKG4R9tjxUKjQeJvb4WmfjLksGFyvstHRfYbBC7XNz, Piece ID: VWS32XVNWH4R5PZDDV4FGP5ONZQZNLMU5YYB7QYS43GDJSDFX2OQ) context deadline exceeded ======LOGS====== Dec 4 17:22:34.320746327 (24h27m10.791957066s) [upload=21507]: Starting upload Dec 4 17:22:34.320860715 (24h27m10.792071454s) [upload=21507]: Done starting upload Dec 4 17:22:34.321208968 (24h27m10.792419707s) [upload=21507]: Uploading segments... Dec 4 17:22:34.321325259 (24h27m10.792536018s) [upload=21507]: Got next segment. Inline: false Dec 4 17:22:34.321394982 (24h27m10.792605711s) [upload=21507]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 17:22:34.321401375 (24h27m10.792612114s) [upload=21507]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 17:22:34.827776318 (24h27m11.298987057s) [seg_pos={0 0}, upload=21507]: Begin upload segment... Dec 4 17:22:34.827838226 (24h27m11.299048966s) [seg_pos={0 0}, upload=21507]: Done begin upload segment. Dec 4 17:22:34.827877361 (24h27m11.29908811s) [noise=true , address=77.90.31.14:4661 , piece_id=D5L5WWE5DGEYKP5M2NM74EGSX5BLEP4DX5OBM2Z4Q2QTCSGA4WXQ, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.827887630 (24h27m11.299098359s) [noise=true , address=77.90.31.21:4878 , piece_id=GHU3SZUB5WAMW37CYCUFJ6HOPMXRNZGIJYDC4DJ3WOHV7YYWCVEA, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.828727861 (24h27m11.2999386s) [noise=true , address=145.79.227.29:4942 , piece_id=QOOXF6G2OGBNJLA64B2YNVVJKAZSP6HM5ZIEI4ZFJKWZUSQAR3XA, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.828821058 (24h27m11.300031807s) [noise=true , address=145.79.227.16:4910 , piece_id=36JF4CMQH5JO66BY5A53QNZQ6RIFTHZ6MD2QLBGHLHML5VBFGK7A, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.829248923 (24h27m11.300459652s) [noise=true , address=145.79.227.43:4157 , piece_id=VWS32XVNWH4R5PZDDV4FGP5ONZQZNLMU5YYB7QYS43GDJSDFX2OQ, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.829618296 (24h27m11.300829035s) [noise=true , address=145.79.227.38:4978 , piece_id=JLYEPL2WLUJIIUHI3FSOWIT5QG3ZQG3NNOZZ3TNMAOEWLZJNN6AA, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.830015262 (24h27m11.301226001s) [noise=true , address=77.90.31.19:4843 , piece_id=36QLYXSVN6L2FDKVX4WS7MRTK54XLB2JUTW2NKHCE7E6KZGTLUKQ, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.830278002 (24h27m11.301488741s) [noise=true , address=145.79.227.21:4475 , piece_id=YZRHRIA7WWLZKDX3BZ2FSUOKJ3MX7D7Q2GB6CHKVQKGXPQCUNC6A, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.830415063 (24h27m11.301625802s) [noise=true , address=145.79.227.8:4329 , piece_id=FBD34B6US36TX6YHN5UEBKOWIVOOP4G7B6SSYKN6Q3J5YNE7OBHA, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.830692902 (24h27m11.301903641s) [noise=true , address=145.79.227.9:4366 , piece_id=GDCC3W66FFB27QD5MRCG2MBJXZBWXRIWG7A4L3XMFTIRNN2NYPUQ, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.830811337 (24h27m11.302022076s) [noise=true , address=77.90.31.10:4520 , piece_id=DGMWUPIGVLYJ4IH2CXCHA3U7FHDJF7APG73XN6TUEWRBB3WZOURQ, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.831349433 (24h27m11.302560172s) [noise=true , address=77.90.31.18:4807 , piece_id=WJ3AOAEE3IG3T3QQRFSKTOJCYVST4263Y5UN6XVOVTSO5ALOVTCQ, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.831805731 (24h27m11.30301647s) [noise=true , address=145.79.227.43:4158 , piece_id=74VKW6UWNPLSDKXJ6AG6TTUSDBCW4K4BAGMFIDNIV55AIWRBVB2Q, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.832172379 (24h27m11.303383118s) [noise=true , address=145.79.227.39:4014 , piece_id=ZLJ7L4OXXXNMJP2NTDU5HNE7NDCFL6JNWLXB6KVZHHFZXF5AT4CA, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.833230595 (24h27m11.304441334s) [noise=true , address=145.79.227.34:4905 , piece_id=ULLE3QB2QSCSYQWZ4OSBDJHC3X7AOHJGRWRA4DUXAMOYBDJRYWRQ, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.833972006 (24h27m11.305182755s) [noise=true , address=145.79.227.10:4401 , piece_id=TLP7B6MNA7W35UJQKW5SW6F43YXXMUFXOUKW4DKSOWOPIKLSSVUA, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:35.930150729 (24h27m12.401361488s) [noise=true , address=77.90.31.19:4843 , piece_id=36QLYXSVN6L2FDKVX4WS7MRTK54XLB2JUTW2NKHCE7E6KZGTLUKQ, seg_pos={0 0}, upload=21507]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: storage node overloaded, request limit: 2000 Dec 4 17:22:36.479057771 (24h27m12.9502685s) [upload=21507]: Next returned nil segment Dec 4 17:22:36.479122765 (24h27m12.950333494s) [upload=21507]: Waiting for error group managing segments... Dec 4 17:22:36.876009453 (24h27m13.347220192s) [noise=true , address=77.90.31.21:4878 , piece_id=GHU3SZUB5WAMW37CYCUFJ6HOPMXRNZGIJYDC4DJ3WOHV7YYWCVEA, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:36.891015988 (24h27m13.362226717s) [noise=true , address=77.90.31.14:4661 , piece_id=D5L5WWE5DGEYKP5M2NM74EGSX5BLEP4DX5OBM2Z4Q2QTCSGA4WXQ, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:36.943731919 (24h27m13.414942658s) [noise=true , address=77.90.31.10:4520 , piece_id=DGMWUPIGVLYJ4IH2CXCHA3U7FHDJF7APG73XN6TUEWRBB3WZOURQ, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:36.981669560 (24h27m13.452880289s) [noise=true , address=145.79.227.16:4910 , piece_id=36JF4CMQH5JO66BY5A53QNZQ6RIFTHZ6MD2QLBGHLHML5VBFGK7A, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:36.981785620 (24h27m13.452996359s) [noise=true , address=145.79.227.21:4475 , piece_id=YZRHRIA7WWLZKDX3BZ2FSUOKJ3MX7D7Q2GB6CHKVQKGXPQCUNC6A, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:36.984955717 (24h27m13.456166456s) [noise=true , address=77.90.31.18:4807 , piece_id=WJ3AOAEE3IG3T3QQRFSKTOJCYVST4263Y5UN6XVOVTSO5ALOVTCQ, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:36.994452571 (24h27m13.4656633s) [noise=true , address=145.79.227.34:4905 , piece_id=ULLE3QB2QSCSYQWZ4OSBDJHC3X7AOHJGRWRA4DUXAMOYBDJRYWRQ, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:37.011055396 (24h27m13.482266125s) [noise=true , address=145.79.227.38:4978 , piece_id=JLYEPL2WLUJIIUHI3FSOWIT5QG3ZQG3NNOZZ3TNMAOEWLZJNN6AA, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:37.021499513 (24h27m13.492710252s) [noise=true , address=145.79.227.8:4329 , piece_id=FBD34B6US36TX6YHN5UEBKOWIVOOP4G7B6SSYKN6Q3J5YNE7OBHA, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:37.043392031 (24h27m13.51460277s) [noise=true , address=145.79.227.39:4014 , piece_id=ZLJ7L4OXXXNMJP2NTDU5HNE7NDCFL6JNWLXB6KVZHHFZXF5AT4CA, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:37.047232213 (24h27m13.518442952s) [noise=true , address=145.79.227.29:4942 , piece_id=QOOXF6G2OGBNJLA64B2YNVVJKAZSP6HM5ZIEI4ZFJKWZUSQAR3XA, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:37.315831267 (24h27m13.787042006s) [noise=true , address=145.79.227.10:4401 , piece_id=TLP7B6MNA7W35UJQKW5SW6F43YXXMUFXOUKW4DKSOWOPIKLSSVUA, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:37.332403453 (24h27m13.803614192s) [noise=true , address=145.79.227.43:4157 , piece_id=VWS32XVNWH4R5PZDDV4FGP5ONZQZNLMU5YYB7QYS43GDJSDFX2OQ, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:37.332432619 (24h27m13.803643358s) [seg_pos={0 0}, upload=21507]: Segment reached optimal threshold of 13 pieces. Dec 4 17:22:37.332650384 (24h27m13.803861123s) [noise=true , address=145.79.227.9:4366 , piece_id=GDCC3W66FFB27QD5MRCG2MBJXZBWXRIWG7A4L3XMFTIRNN2NYPUQ, seg_pos={0 0}, upload=21507]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy): protocol: context canceled Dec 4 17:22:37.332660022 (24h27m13.803870751s) [noise=true , address=145.79.227.9:4366 , piece_id=GDCC3W66FFB27QD5MRCG2MBJXZBWXRIWG7A4L3XMFTIRNN2NYPUQ, seg_pos={0 0}, upload=21507]: Long tail cancellation due to optimal threshold Dec 4 17:22:37.332942179 (24h27m13.804152938s) [noise=true , address=145.79.227.43:4158 , piece_id=74VKW6UWNPLSDKXJ6AG6TTUSDBCW4K4BAGMFIDNIV55AIWRBVB2Q, seg_pos={0 0}, upload=21507]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vJZrSvPvwebw9V2tTbvkPEmLeLjryBUnq9TL6NwdPsHU9xcq2): protocol: expected piece hash; context canceled; EOF Dec 4 17:22:37.332954993 (24h27m13.804165732s) [noise=true , address=145.79.227.43:4158 , piece_id=74VKW6UWNPLSDKXJ6AG6TTUSDBCW4K4BAGMFIDNIV55AIWRBVB2Q, seg_pos={0 0}, upload=21507]: Long tail cancellation due to optimal threshold Dec 4 17:22:37.332976635 (24h27m13.804187384s) [seg_pos={0 0}, upload=21507]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 4 17:22:37.333114427 (24h27m13.804325166s) [upload=21507]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 4 17:22:37.333121821 (24h27m13.80433255s) [upload=21507]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 4 17:22:37.958438444 (24h27m14.429649173s) [upload=21507]: Done uploading segments... Dec 4 17:22:37.958456208 (24h27m14.429666937s) [upload=21507]: Upload finished. err: <nil> Duration: 5.693984986s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 5.922682901s trace: https://jaeger.dev.storj.io/trace/400044794bd63bf0 FAIL (2025-12-04 17:37:56 UTC) orbiter warning: read tcp 188.166.225.170:54202->45.38.24.2:443: read: connection reset by peer Duration: 5.922710965s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 5.664945086s trace: https://jaeger.dev.storj.io/trace/400044794bd63bf0 FAIL (2025-12-04 17:37:36 UTC) orbiter warning: read tcp 188.166.225.170:54202->45.38.24.2:443: read: connection reset by peer Duration: 5.664987107s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.............X..............................................] duration: 3.366043339s trace: https://jaeger.dev.storj.io/trace/1f3e769edbb82b7a FAIL (2025-12-04 17:37:28 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.366077613s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 5.843934675s trace: https://jaeger.dev.storj.io/trace/400044794bd63bf0 FAIL (2025-12-04 17:37:03 UTC) orbiter warning: read tcp 188.166.225.170:54202->45.38.24.2:443: read: connection reset by peer Duration: 5.843968178s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 5.839417111s trace: https://jaeger.dev.storj.io/trace/6d15aafc5ce7b199 FAIL (2025-12-04 17:36:47 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, Piece ID: FBD34B6US36TX6YHN5UEBKOWIVOOP4G7B6SSYKN6Q3J5YNE7OBHA) context deadline exceeded; piecestore: (Node ID: 12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9, Piece ID: ULLE3QB2QSCSYQWZ4OSBDJHC3X7AOHJGRWRA4DUXAMOYBDJRYWRQ) context deadline exceeded; piecestore: (Node ID: 1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, Piece ID: TLP7B6MNA7W35UJQKW5SW6F43YXXMUFXOUKW4DKSOWOPIKLSSVUA) context deadline exceeded; piecestore: (Node ID: 17o881pUMuG7K2dv83k7RkrJCrmM7kgUiWesgam4GEw21KVUn7, Piece ID: QOOXF6G2OGBNJLA64B2YNVVJKAZSP6HM5ZIEI4ZFJKWZUSQAR3XA) context deadline exceeded; piecestore: (Node ID: 1ZrzADk2mKG4R9tjxUKjQeJvb4WmfjLksGFyvstHRfYbBC7XNz, Piece ID: VWS32XVNWH4R5PZDDV4FGP5ONZQZNLMU5YYB7QYS43GDJSDFX2OQ) context deadline exceeded ======LOGS====== Dec 4 17:22:34.320746327 (24h27m10.791957066s) [upload=21507]: Starting upload Dec 4 17:22:34.320860715 (24h27m10.792071454s) [upload=21507]: Done starting upload Dec 4 17:22:34.321208968 (24h27m10.792419707s) [upload=21507]: Uploading segments... Dec 4 17:22:34.321325259 (24h27m10.792536018s) [upload=21507]: Got next segment. Inline: false Dec 4 17:22:34.321394982 (24h27m10.792605711s) [upload=21507]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 17:22:34.321401375 (24h27m10.792612114s) [upload=21507]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 17:22:34.827776318 (24h27m11.298987057s) [seg_pos={0 0}, upload=21507]: Begin upload segment... Dec 4 17:22:34.827838226 (24h27m11.299048966s) [seg_pos={0 0}, upload=21507]: Done begin upload segment. Dec 4 17:22:34.827877361 (24h27m11.29908811s) [noise=true , address=77.90.31.14:4661 , piece_id=D5L5WWE5DGEYKP5M2NM74EGSX5BLEP4DX5OBM2Z4Q2QTCSGA4WXQ, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.827887630 (24h27m11.299098359s) [noise=true , address=77.90.31.21:4878 , piece_id=GHU3SZUB5WAMW37CYCUFJ6HOPMXRNZGIJYDC4DJ3WOHV7YYWCVEA, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.828727861 (24h27m11.2999386s) [noise=true , address=145.79.227.29:4942 , piece_id=QOOXF6G2OGBNJLA64B2YNVVJKAZSP6HM5ZIEI4ZFJKWZUSQAR3XA, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.828821058 (24h27m11.300031807s) [noise=true , address=145.79.227.16:4910 , piece_id=36JF4CMQH5JO66BY5A53QNZQ6RIFTHZ6MD2QLBGHLHML5VBFGK7A, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.829248923 (24h27m11.300459652s) [noise=true , address=145.79.227.43:4157 , piece_id=VWS32XVNWH4R5PZDDV4FGP5ONZQZNLMU5YYB7QYS43GDJSDFX2OQ, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.829618296 (24h27m11.300829035s) [noise=true , address=145.79.227.38:4978 , piece_id=JLYEPL2WLUJIIUHI3FSOWIT5QG3ZQG3NNOZZ3TNMAOEWLZJNN6AA, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.830015262 (24h27m11.301226001s) [noise=true , address=77.90.31.19:4843 , piece_id=36QLYXSVN6L2FDKVX4WS7MRTK54XLB2JUTW2NKHCE7E6KZGTLUKQ, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.830278002 (24h27m11.301488741s) [noise=true , address=145.79.227.21:4475 , piece_id=YZRHRIA7WWLZKDX3BZ2FSUOKJ3MX7D7Q2GB6CHKVQKGXPQCUNC6A, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.830415063 (24h27m11.301625802s) [noise=true , address=145.79.227.8:4329 , piece_id=FBD34B6US36TX6YHN5UEBKOWIVOOP4G7B6SSYKN6Q3J5YNE7OBHA, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.830692902 (24h27m11.301903641s) [noise=true , address=145.79.227.9:4366 , piece_id=GDCC3W66FFB27QD5MRCG2MBJXZBWXRIWG7A4L3XMFTIRNN2NYPUQ, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.830811337 (24h27m11.302022076s) [noise=true , address=77.90.31.10:4520 , piece_id=DGMWUPIGVLYJ4IH2CXCHA3U7FHDJF7APG73XN6TUEWRBB3WZOURQ, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.831349433 (24h27m11.302560172s) [noise=true , address=77.90.31.18:4807 , piece_id=WJ3AOAEE3IG3T3QQRFSKTOJCYVST4263Y5UN6XVOVTSO5ALOVTCQ, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.831805731 (24h27m11.30301647s) [noise=true , address=145.79.227.43:4158 , piece_id=74VKW6UWNPLSDKXJ6AG6TTUSDBCW4K4BAGMFIDNIV55AIWRBVB2Q, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.832172379 (24h27m11.303383118s) [noise=true , address=145.79.227.39:4014 , piece_id=ZLJ7L4OXXXNMJP2NTDU5HNE7NDCFL6JNWLXB6KVZHHFZXF5AT4CA, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.833230595 (24h27m11.304441334s) [noise=true , address=145.79.227.34:4905 , piece_id=ULLE3QB2QSCSYQWZ4OSBDJHC3X7AOHJGRWRA4DUXAMOYBDJRYWRQ, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.833972006 (24h27m11.305182755s) [noise=true , address=145.79.227.10:4401 , piece_id=TLP7B6MNA7W35UJQKW5SW6F43YXXMUFXOUKW4DKSOWOPIKLSSVUA, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:35.930150729 (24h27m12.401361488s) [noise=true , address=77.90.31.19:4843 , piece_id=36QLYXSVN6L2FDKVX4WS7MRTK54XLB2JUTW2NKHCE7E6KZGTLUKQ, seg_pos={0 0}, upload=21507]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: storage node overloaded, request limit: 2000 Dec 4 17:22:36.479057771 (24h27m12.9502685s) [upload=21507]: Next returned nil segment Dec 4 17:22:36.479122765 (24h27m12.950333494s) [upload=21507]: Waiting for error group managing segments... Dec 4 17:22:36.876009453 (24h27m13.347220192s) [noise=true , address=77.90.31.21:4878 , piece_id=GHU3SZUB5WAMW37CYCUFJ6HOPMXRNZGIJYDC4DJ3WOHV7YYWCVEA, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:36.891015988 (24h27m13.362226717s) [noise=true , address=77.90.31.14:4661 , piece_id=D5L5WWE5DGEYKP5M2NM74EGSX5BLEP4DX5OBM2Z4Q2QTCSGA4WXQ, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:36.943731919 (24h27m13.414942658s) [noise=true , address=77.90.31.10:4520 , piece_id=DGMWUPIGVLYJ4IH2CXCHA3U7FHDJF7APG73XN6TUEWRBB3WZOURQ, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:36.981669560 (24h27m13.452880289s) [noise=true , address=145.79.227.16:4910 , piece_id=36JF4CMQH5JO66BY5A53QNZQ6RIFTHZ6MD2QLBGHLHML5VBFGK7A, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:36.981785620 (24h27m13.452996359s) [noise=true , address=145.79.227.21:4475 , piece_id=YZRHRIA7WWLZKDX3BZ2FSUOKJ3MX7D7Q2GB6CHKVQKGXPQCUNC6A, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:36.984955717 (24h27m13.456166456s) [noise=true , address=77.90.31.18:4807 , piece_id=WJ3AOAEE3IG3T3QQRFSKTOJCYVST4263Y5UN6XVOVTSO5ALOVTCQ, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:36.994452571 (24h27m13.4656633s) [noise=true , address=145.79.227.34:4905 , piece_id=ULLE3QB2QSCSYQWZ4OSBDJHC3X7AOHJGRWRA4DUXAMOYBDJRYWRQ, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:37.011055396 (24h27m13.482266125s) [noise=true , address=145.79.227.38:4978 , piece_id=JLYEPL2WLUJIIUHI3FSOWIT5QG3ZQG3NNOZZ3TNMAOEWLZJNN6AA, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:37.021499513 (24h27m13.492710252s) [noise=true , address=145.79.227.8:4329 , piece_id=FBD34B6US36TX6YHN5UEBKOWIVOOP4G7B6SSYKN6Q3J5YNE7OBHA, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:37.043392031 (24h27m13.51460277s) [noise=true , address=145.79.227.39:4014 , piece_id=ZLJ7L4OXXXNMJP2NTDU5HNE7NDCFL6JNWLXB6KVZHHFZXF5AT4CA, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:37.047232213 (24h27m13.518442952s) [noise=true , address=145.79.227.29:4942 , piece_id=QOOXF6G2OGBNJLA64B2YNVVJKAZSP6HM5ZIEI4ZFJKWZUSQAR3XA, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:37.315831267 (24h27m13.787042006s) [noise=true , address=145.79.227.10:4401 , piece_id=TLP7B6MNA7W35UJQKW5SW6F43YXXMUFXOUKW4DKSOWOPIKLSSVUA, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:37.332403453 (24h27m13.803614192s) [noise=true , address=145.79.227.43:4157 , piece_id=VWS32XVNWH4R5PZDDV4FGP5ONZQZNLMU5YYB7QYS43GDJSDFX2OQ, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:37.332432619 (24h27m13.803643358s) [seg_pos={0 0}, upload=21507]: Segment reached optimal threshold of 13 pieces. Dec 4 17:22:37.332650384 (24h27m13.803861123s) [noise=true , address=145.79.227.9:4366 , piece_id=GDCC3W66FFB27QD5MRCG2MBJXZBWXRIWG7A4L3XMFTIRNN2NYPUQ, seg_pos={0 0}, upload=21507]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy): protocol: context canceled Dec 4 17:22:37.332660022 (24h27m13.803870751s) [noise=true , address=145.79.227.9:4366 , piece_id=GDCC3W66FFB27QD5MRCG2MBJXZBWXRIWG7A4L3XMFTIRNN2NYPUQ, seg_pos={0 0}, upload=21507]: Long tail cancellation due to optimal threshold Dec 4 17:22:37.332942179 (24h27m13.804152938s) [noise=true , address=145.79.227.43:4158 , piece_id=74VKW6UWNPLSDKXJ6AG6TTUSDBCW4K4BAGMFIDNIV55AIWRBVB2Q, seg_pos={0 0}, upload=21507]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vJZrSvPvwebw9V2tTbvkPEmLeLjryBUnq9TL6NwdPsHU9xcq2): protocol: expected piece hash; context canceled; EOF Dec 4 17:22:37.332954993 (24h27m13.804165732s) [noise=true , address=145.79.227.43:4158 , piece_id=74VKW6UWNPLSDKXJ6AG6TTUSDBCW4K4BAGMFIDNIV55AIWRBVB2Q, seg_pos={0 0}, upload=21507]: Long tail cancellation due to optimal threshold Dec 4 17:22:37.332976635 (24h27m13.804187384s) [seg_pos={0 0}, upload=21507]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 4 17:22:37.333114427 (24h27m13.804325166s) [upload=21507]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 4 17:22:37.333121821 (24h27m13.80433255s) [upload=21507]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 4 17:22:37.958438444 (24h27m14.429649173s) [upload=21507]: Done uploading segments... Dec 4 17:22:37.958456208 (24h27m14.429666937s) [upload=21507]: Upload finished. err: <nil> Duration: 5.839448s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..............X.............................................] duration: 3.445792777s trace: https://jaeger.dev.storj.io/trace/1f3e769edbb82b7a FAIL (2025-12-04 17:36:39 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.445824096s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 5.665575696s trace: https://jaeger.dev.storj.io/trace/6d15aafc5ce7b199 FAIL (2025-12-04 17:36:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, Piece ID: FBD34B6US36TX6YHN5UEBKOWIVOOP4G7B6SSYKN6Q3J5YNE7OBHA) context deadline exceeded; piecestore: (Node ID: 12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9, Piece ID: ULLE3QB2QSCSYQWZ4OSBDJHC3X7AOHJGRWRA4DUXAMOYBDJRYWRQ) context deadline exceeded; piecestore: (Node ID: 1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, Piece ID: TLP7B6MNA7W35UJQKW5SW6F43YXXMUFXOUKW4DKSOWOPIKLSSVUA) context deadline exceeded; piecestore: (Node ID: 17o881pUMuG7K2dv83k7RkrJCrmM7kgUiWesgam4GEw21KVUn7, Piece ID: QOOXF6G2OGBNJLA64B2YNVVJKAZSP6HM5ZIEI4ZFJKWZUSQAR3XA) context deadline exceeded; piecestore: (Node ID: 1ZrzADk2mKG4R9tjxUKjQeJvb4WmfjLksGFyvstHRfYbBC7XNz, Piece ID: VWS32XVNWH4R5PZDDV4FGP5ONZQZNLMU5YYB7QYS43GDJSDFX2OQ) context deadline exceeded ======LOGS====== Dec 4 17:22:34.320746327 (24h27m10.791957066s) [upload=21507]: Starting upload Dec 4 17:22:34.320860715 (24h27m10.792071454s) [upload=21507]: Done starting upload Dec 4 17:22:34.321208968 (24h27m10.792419707s) [upload=21507]: Uploading segments... Dec 4 17:22:34.321325259 (24h27m10.792536018s) [upload=21507]: Got next segment. Inline: false Dec 4 17:22:34.321394982 (24h27m10.792605711s) [upload=21507]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 17:22:34.321401375 (24h27m10.792612114s) [upload=21507]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 17:22:34.827776318 (24h27m11.298987057s) [seg_pos={0 0}, upload=21507]: Begin upload segment... Dec 4 17:22:34.827838226 (24h27m11.299048966s) [seg_pos={0 0}, upload=21507]: Done begin upload segment. Dec 4 17:22:34.827877361 (24h27m11.29908811s) [noise=true , address=77.90.31.14:4661 , piece_id=D5L5WWE5DGEYKP5M2NM74EGSX5BLEP4DX5OBM2Z4Q2QTCSGA4WXQ, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.827887630 (24h27m11.299098359s) [noise=true , address=77.90.31.21:4878 , piece_id=GHU3SZUB5WAMW37CYCUFJ6HOPMXRNZGIJYDC4DJ3WOHV7YYWCVEA, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.828727861 (24h27m11.2999386s) [noise=true , address=145.79.227.29:4942 , piece_id=QOOXF6G2OGBNJLA64B2YNVVJKAZSP6HM5ZIEI4ZFJKWZUSQAR3XA, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.828821058 (24h27m11.300031807s) [noise=true , address=145.79.227.16:4910 , piece_id=36JF4CMQH5JO66BY5A53QNZQ6RIFTHZ6MD2QLBGHLHML5VBFGK7A, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.829248923 (24h27m11.300459652s) [noise=true , address=145.79.227.43:4157 , piece_id=VWS32XVNWH4R5PZDDV4FGP5ONZQZNLMU5YYB7QYS43GDJSDFX2OQ, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.829618296 (24h27m11.300829035s) [noise=true , address=145.79.227.38:4978 , piece_id=JLYEPL2WLUJIIUHI3FSOWIT5QG3ZQG3NNOZZ3TNMAOEWLZJNN6AA, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.830015262 (24h27m11.301226001s) [noise=true , address=77.90.31.19:4843 , piece_id=36QLYXSVN6L2FDKVX4WS7MRTK54XLB2JUTW2NKHCE7E6KZGTLUKQ, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.830278002 (24h27m11.301488741s) [noise=true , address=145.79.227.21:4475 , piece_id=YZRHRIA7WWLZKDX3BZ2FSUOKJ3MX7D7Q2GB6CHKVQKGXPQCUNC6A, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.830415063 (24h27m11.301625802s) [noise=true , address=145.79.227.8:4329 , piece_id=FBD34B6US36TX6YHN5UEBKOWIVOOP4G7B6SSYKN6Q3J5YNE7OBHA, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.830692902 (24h27m11.301903641s) [noise=true , address=145.79.227.9:4366 , piece_id=GDCC3W66FFB27QD5MRCG2MBJXZBWXRIWG7A4L3XMFTIRNN2NYPUQ, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.830811337 (24h27m11.302022076s) [noise=true , address=77.90.31.10:4520 , piece_id=DGMWUPIGVLYJ4IH2CXCHA3U7FHDJF7APG73XN6TUEWRBB3WZOURQ, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.831349433 (24h27m11.302560172s) [noise=true , address=77.90.31.18:4807 , piece_id=WJ3AOAEE3IG3T3QQRFSKTOJCYVST4263Y5UN6XVOVTSO5ALOVTCQ, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.831805731 (24h27m11.30301647s) [noise=true , address=145.79.227.43:4158 , piece_id=74VKW6UWNPLSDKXJ6AG6TTUSDBCW4K4BAGMFIDNIV55AIWRBVB2Q, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.832172379 (24h27m11.303383118s) [noise=true , address=145.79.227.39:4014 , piece_id=ZLJ7L4OXXXNMJP2NTDU5HNE7NDCFL6JNWLXB6KVZHHFZXF5AT4CA, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.833230595 (24h27m11.304441334s) [noise=true , address=145.79.227.34:4905 , piece_id=ULLE3QB2QSCSYQWZ4OSBDJHC3X7AOHJGRWRA4DUXAMOYBDJRYWRQ, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.833972006 (24h27m11.305182755s) [noise=true , address=145.79.227.10:4401 , piece_id=TLP7B6MNA7W35UJQKW5SW6F43YXXMUFXOUKW4DKSOWOPIKLSSVUA, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:35.930150729 (24h27m12.401361488s) [noise=true , address=77.90.31.19:4843 , piece_id=36QLYXSVN6L2FDKVX4WS7MRTK54XLB2JUTW2NKHCE7E6KZGTLUKQ, seg_pos={0 0}, upload=21507]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: storage node overloaded, request limit: 2000 Dec 4 17:22:36.479057771 (24h27m12.9502685s) [upload=21507]: Next returned nil segment Dec 4 17:22:36.479122765 (24h27m12.950333494s) [upload=21507]: Waiting for error group managing segments... Dec 4 17:22:36.876009453 (24h27m13.347220192s) [noise=true , address=77.90.31.21:4878 , piece_id=GHU3SZUB5WAMW37CYCUFJ6HOPMXRNZGIJYDC4DJ3WOHV7YYWCVEA, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:36.891015988 (24h27m13.362226717s) [noise=true , address=77.90.31.14:4661 , piece_id=D5L5WWE5DGEYKP5M2NM74EGSX5BLEP4DX5OBM2Z4Q2QTCSGA4WXQ, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:36.943731919 (24h27m13.414942658s) [noise=true , address=77.90.31.10:4520 , piece_id=DGMWUPIGVLYJ4IH2CXCHA3U7FHDJF7APG73XN6TUEWRBB3WZOURQ, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:36.981669560 (24h27m13.452880289s) [noise=true , address=145.79.227.16:4910 , piece_id=36JF4CMQH5JO66BY5A53QNZQ6RIFTHZ6MD2QLBGHLHML5VBFGK7A, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:36.981785620 (24h27m13.452996359s) [noise=true , address=145.79.227.21:4475 , piece_id=YZRHRIA7WWLZKDX3BZ2FSUOKJ3MX7D7Q2GB6CHKVQKGXPQCUNC6A, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:36.984955717 (24h27m13.456166456s) [noise=true , address=77.90.31.18:4807 , piece_id=WJ3AOAEE3IG3T3QQRFSKTOJCYVST4263Y5UN6XVOVTSO5ALOVTCQ, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:36.994452571 (24h27m13.4656633s) [noise=true , address=145.79.227.34:4905 , piece_id=ULLE3QB2QSCSYQWZ4OSBDJHC3X7AOHJGRWRA4DUXAMOYBDJRYWRQ, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:37.011055396 (24h27m13.482266125s) [noise=true , address=145.79.227.38:4978 , piece_id=JLYEPL2WLUJIIUHI3FSOWIT5QG3ZQG3NNOZZ3TNMAOEWLZJNN6AA, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:37.021499513 (24h27m13.492710252s) [noise=true , address=145.79.227.8:4329 , piece_id=FBD34B6US36TX6YHN5UEBKOWIVOOP4G7B6SSYKN6Q3J5YNE7OBHA, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:37.043392031 (24h27m13.51460277s) [noise=true , address=145.79.227.39:4014 , piece_id=ZLJ7L4OXXXNMJP2NTDU5HNE7NDCFL6JNWLXB6KVZHHFZXF5AT4CA, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:37.047232213 (24h27m13.518442952s) [noise=true , address=145.79.227.29:4942 , piece_id=QOOXF6G2OGBNJLA64B2YNVVJKAZSP6HM5ZIEI4ZFJKWZUSQAR3XA, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:37.315831267 (24h27m13.787042006s) [noise=true , address=145.79.227.10:4401 , piece_id=TLP7B6MNA7W35UJQKW5SW6F43YXXMUFXOUKW4DKSOWOPIKLSSVUA, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:37.332403453 (24h27m13.803614192s) [noise=true , address=145.79.227.43:4157 , piece_id=VWS32XVNWH4R5PZDDV4FGP5ONZQZNLMU5YYB7QYS43GDJSDFX2OQ, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:37.332432619 (24h27m13.803643358s) [seg_pos={0 0}, upload=21507]: Segment reached optimal threshold of 13 pieces. Dec 4 17:22:37.332650384 (24h27m13.803861123s) [noise=true , address=145.79.227.9:4366 , piece_id=GDCC3W66FFB27QD5MRCG2MBJXZBWXRIWG7A4L3XMFTIRNN2NYPUQ, seg_pos={0 0}, upload=21507]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy): protocol: context canceled Dec 4 17:22:37.332660022 (24h27m13.803870751s) [noise=true , address=145.79.227.9:4366 , piece_id=GDCC3W66FFB27QD5MRCG2MBJXZBWXRIWG7A4L3XMFTIRNN2NYPUQ, seg_pos={0 0}, upload=21507]: Long tail cancellation due to optimal threshold Dec 4 17:22:37.332942179 (24h27m13.804152938s) [noise=true , address=145.79.227.43:4158 , piece_id=74VKW6UWNPLSDKXJ6AG6TTUSDBCW4K4BAGMFIDNIV55AIWRBVB2Q, seg_pos={0 0}, upload=21507]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vJZrSvPvwebw9V2tTbvkPEmLeLjryBUnq9TL6NwdPsHU9xcq2): protocol: expected piece hash; context canceled; EOF Dec 4 17:22:37.332954993 (24h27m13.804165732s) [noise=true , address=145.79.227.43:4158 , piece_id=74VKW6UWNPLSDKXJ6AG6TTUSDBCW4K4BAGMFIDNIV55AIWRBVB2Q, seg_pos={0 0}, upload=21507]: Long tail cancellation due to optimal threshold Dec 4 17:22:37.332976635 (24h27m13.804187384s) [seg_pos={0 0}, upload=21507]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 4 17:22:37.333114427 (24h27m13.804325166s) [upload=21507]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 4 17:22:37.333121821 (24h27m13.80433255s) [upload=21507]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 4 17:22:37.958438444 (24h27m14.429649173s) [upload=21507]: Done uploading segments... Dec 4 17:22:37.958456208 (24h27m14.429666937s) [upload=21507]: Upload finished. err: <nil> Duration: 5.66560902s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 5.753092354s trace: https://jaeger.dev.storj.io/trace/400044794bd63bf0 FAIL (2025-12-04 17:36:20 UTC) orbiter warning: read tcp 188.166.225.170:54202->45.38.24.2:443: read: connection reset by peer Duration: 5.753126619s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...............X............................................] duration: 3.44002236s trace: https://jaeger.dev.storj.io/trace/1f3e769edbb82b7a FAIL (2025-12-04 17:36:19 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.440090649s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [................X...........................................] duration: 3.437325576s trace: https://jaeger.dev.storj.io/trace/1f3e769edbb82b7a FAIL (2025-12-04 17:35:51 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.437351336s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 5.788253141s trace: https://jaeger.dev.storj.io/trace/400044794bd63bf0 FAIL (2025-12-04 17:35:34 UTC) orbiter warning: read tcp 188.166.225.170:54202->45.38.24.2:443: read: connection reset by peer Duration: 5.788280572s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 5.704321316s trace: https://jaeger.dev.storj.io/trace/6d15aafc5ce7b199 FAIL (2025-12-04 17:35:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, Piece ID: FBD34B6US36TX6YHN5UEBKOWIVOOP4G7B6SSYKN6Q3J5YNE7OBHA) context deadline exceeded; piecestore: (Node ID: 12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9, Piece ID: ULLE3QB2QSCSYQWZ4OSBDJHC3X7AOHJGRWRA4DUXAMOYBDJRYWRQ) context deadline exceeded; piecestore: (Node ID: 1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, Piece ID: TLP7B6MNA7W35UJQKW5SW6F43YXXMUFXOUKW4DKSOWOPIKLSSVUA) context deadline exceeded; piecestore: (Node ID: 17o881pUMuG7K2dv83k7RkrJCrmM7kgUiWesgam4GEw21KVUn7, Piece ID: QOOXF6G2OGBNJLA64B2YNVVJKAZSP6HM5ZIEI4ZFJKWZUSQAR3XA) context deadline exceeded; piecestore: (Node ID: 1ZrzADk2mKG4R9tjxUKjQeJvb4WmfjLksGFyvstHRfYbBC7XNz, Piece ID: VWS32XVNWH4R5PZDDV4FGP5ONZQZNLMU5YYB7QYS43GDJSDFX2OQ) context deadline exceeded ======LOGS====== Dec 4 17:22:34.320746327 (24h27m10.791957066s) [upload=21507]: Starting upload Dec 4 17:22:34.320860715 (24h27m10.792071454s) [upload=21507]: Done starting upload Dec 4 17:22:34.321208968 (24h27m10.792419707s) [upload=21507]: Uploading segments... Dec 4 17:22:34.321325259 (24h27m10.792536018s) [upload=21507]: Got next segment. Inline: false Dec 4 17:22:34.321394982 (24h27m10.792605711s) [upload=21507]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 17:22:34.321401375 (24h27m10.792612114s) [upload=21507]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 17:22:34.827776318 (24h27m11.298987057s) [seg_pos={0 0}, upload=21507]: Begin upload segment... Dec 4 17:22:34.827838226 (24h27m11.299048966s) [seg_pos={0 0}, upload=21507]: Done begin upload segment. Dec 4 17:22:34.827877361 (24h27m11.29908811s) [noise=true , address=77.90.31.14:4661 , piece_id=D5L5WWE5DGEYKP5M2NM74EGSX5BLEP4DX5OBM2Z4Q2QTCSGA4WXQ, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.827887630 (24h27m11.299098359s) [noise=true , address=77.90.31.21:4878 , piece_id=GHU3SZUB5WAMW37CYCUFJ6HOPMXRNZGIJYDC4DJ3WOHV7YYWCVEA, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.828727861 (24h27m11.2999386s) [noise=true , address=145.79.227.29:4942 , piece_id=QOOXF6G2OGBNJLA64B2YNVVJKAZSP6HM5ZIEI4ZFJKWZUSQAR3XA, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.828821058 (24h27m11.300031807s) [noise=true , address=145.79.227.16:4910 , piece_id=36JF4CMQH5JO66BY5A53QNZQ6RIFTHZ6MD2QLBGHLHML5VBFGK7A, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.829248923 (24h27m11.300459652s) [noise=true , address=145.79.227.43:4157 , piece_id=VWS32XVNWH4R5PZDDV4FGP5ONZQZNLMU5YYB7QYS43GDJSDFX2OQ, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.829618296 (24h27m11.300829035s) [noise=true , address=145.79.227.38:4978 , piece_id=JLYEPL2WLUJIIUHI3FSOWIT5QG3ZQG3NNOZZ3TNMAOEWLZJNN6AA, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.830015262 (24h27m11.301226001s) [noise=true , address=77.90.31.19:4843 , piece_id=36QLYXSVN6L2FDKVX4WS7MRTK54XLB2JUTW2NKHCE7E6KZGTLUKQ, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.830278002 (24h27m11.301488741s) [noise=true , address=145.79.227.21:4475 , piece_id=YZRHRIA7WWLZKDX3BZ2FSUOKJ3MX7D7Q2GB6CHKVQKGXPQCUNC6A, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.830415063 (24h27m11.301625802s) [noise=true , address=145.79.227.8:4329 , piece_id=FBD34B6US36TX6YHN5UEBKOWIVOOP4G7B6SSYKN6Q3J5YNE7OBHA, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.830692902 (24h27m11.301903641s) [noise=true , address=145.79.227.9:4366 , piece_id=GDCC3W66FFB27QD5MRCG2MBJXZBWXRIWG7A4L3XMFTIRNN2NYPUQ, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.830811337 (24h27m11.302022076s) [noise=true , address=77.90.31.10:4520 , piece_id=DGMWUPIGVLYJ4IH2CXCHA3U7FHDJF7APG73XN6TUEWRBB3WZOURQ, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.831349433 (24h27m11.302560172s) [noise=true , address=77.90.31.18:4807 , piece_id=WJ3AOAEE3IG3T3QQRFSKTOJCYVST4263Y5UN6XVOVTSO5ALOVTCQ, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.831805731 (24h27m11.30301647s) [noise=true , address=145.79.227.43:4158 , piece_id=74VKW6UWNPLSDKXJ6AG6TTUSDBCW4K4BAGMFIDNIV55AIWRBVB2Q, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.832172379 (24h27m11.303383118s) [noise=true , address=145.79.227.39:4014 , piece_id=ZLJ7L4OXXXNMJP2NTDU5HNE7NDCFL6JNWLXB6KVZHHFZXF5AT4CA, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.833230595 (24h27m11.304441334s) [noise=true , address=145.79.227.34:4905 , piece_id=ULLE3QB2QSCSYQWZ4OSBDJHC3X7AOHJGRWRA4DUXAMOYBDJRYWRQ, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.833972006 (24h27m11.305182755s) [noise=true , address=145.79.227.10:4401 , piece_id=TLP7B6MNA7W35UJQKW5SW6F43YXXMUFXOUKW4DKSOWOPIKLSSVUA, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:35.930150729 (24h27m12.401361488s) [noise=true , address=77.90.31.19:4843 , piece_id=36QLYXSVN6L2FDKVX4WS7MRTK54XLB2JUTW2NKHCE7E6KZGTLUKQ, seg_pos={0 0}, upload=21507]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: storage node overloaded, request limit: 2000 Dec 4 17:22:36.479057771 (24h27m12.9502685s) [upload=21507]: Next returned nil segment Dec 4 17:22:36.479122765 (24h27m12.950333494s) [upload=21507]: Waiting for error group managing segments... Dec 4 17:22:36.876009453 (24h27m13.347220192s) [noise=true , address=77.90.31.21:4878 , piece_id=GHU3SZUB5WAMW37CYCUFJ6HOPMXRNZGIJYDC4DJ3WOHV7YYWCVEA, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:36.891015988 (24h27m13.362226717s) [noise=true , address=77.90.31.14:4661 , piece_id=D5L5WWE5DGEYKP5M2NM74EGSX5BLEP4DX5OBM2Z4Q2QTCSGA4WXQ, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:36.943731919 (24h27m13.414942658s) [noise=true , address=77.90.31.10:4520 , piece_id=DGMWUPIGVLYJ4IH2CXCHA3U7FHDJF7APG73XN6TUEWRBB3WZOURQ, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:36.981669560 (24h27m13.452880289s) [noise=true , address=145.79.227.16:4910 , piece_id=36JF4CMQH5JO66BY5A53QNZQ6RIFTHZ6MD2QLBGHLHML5VBFGK7A, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:36.981785620 (24h27m13.452996359s) [noise=true , address=145.79.227.21:4475 , piece_id=YZRHRIA7WWLZKDX3BZ2FSUOKJ3MX7D7Q2GB6CHKVQKGXPQCUNC6A, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:36.984955717 (24h27m13.456166456s) [noise=true , address=77.90.31.18:4807 , piece_id=WJ3AOAEE3IG3T3QQRFSKTOJCYVST4263Y5UN6XVOVTSO5ALOVTCQ, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:36.994452571 (24h27m13.4656633s) [noise=true , address=145.79.227.34:4905 , piece_id=ULLE3QB2QSCSYQWZ4OSBDJHC3X7AOHJGRWRA4DUXAMOYBDJRYWRQ, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:37.011055396 (24h27m13.482266125s) [noise=true , address=145.79.227.38:4978 , piece_id=JLYEPL2WLUJIIUHI3FSOWIT5QG3ZQG3NNOZZ3TNMAOEWLZJNN6AA, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:37.021499513 (24h27m13.492710252s) [noise=true , address=145.79.227.8:4329 , piece_id=FBD34B6US36TX6YHN5UEBKOWIVOOP4G7B6SSYKN6Q3J5YNE7OBHA, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:37.043392031 (24h27m13.51460277s) [noise=true , address=145.79.227.39:4014 , piece_id=ZLJ7L4OXXXNMJP2NTDU5HNE7NDCFL6JNWLXB6KVZHHFZXF5AT4CA, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:37.047232213 (24h27m13.518442952s) [noise=true , address=145.79.227.29:4942 , piece_id=QOOXF6G2OGBNJLA64B2YNVVJKAZSP6HM5ZIEI4ZFJKWZUSQAR3XA, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:37.315831267 (24h27m13.787042006s) [noise=true , address=145.79.227.10:4401 , piece_id=TLP7B6MNA7W35UJQKW5SW6F43YXXMUFXOUKW4DKSOWOPIKLSSVUA, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:37.332403453 (24h27m13.803614192s) [noise=true , address=145.79.227.43:4157 , piece_id=VWS32XVNWH4R5PZDDV4FGP5ONZQZNLMU5YYB7QYS43GDJSDFX2OQ, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:37.332432619 (24h27m13.803643358s) [seg_pos={0 0}, upload=21507]: Segment reached optimal threshold of 13 pieces. Dec 4 17:22:37.332650384 (24h27m13.803861123s) [noise=true , address=145.79.227.9:4366 , piece_id=GDCC3W66FFB27QD5MRCG2MBJXZBWXRIWG7A4L3XMFTIRNN2NYPUQ, seg_pos={0 0}, upload=21507]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy): protocol: context canceled Dec 4 17:22:37.332660022 (24h27m13.803870751s) [noise=true , address=145.79.227.9:4366 , piece_id=GDCC3W66FFB27QD5MRCG2MBJXZBWXRIWG7A4L3XMFTIRNN2NYPUQ, seg_pos={0 0}, upload=21507]: Long tail cancellation due to optimal threshold Dec 4 17:22:37.332942179 (24h27m13.804152938s) [noise=true , address=145.79.227.43:4158 , piece_id=74VKW6UWNPLSDKXJ6AG6TTUSDBCW4K4BAGMFIDNIV55AIWRBVB2Q, seg_pos={0 0}, upload=21507]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vJZrSvPvwebw9V2tTbvkPEmLeLjryBUnq9TL6NwdPsHU9xcq2): protocol: expected piece hash; context canceled; EOF Dec 4 17:22:37.332954993 (24h27m13.804165732s) [noise=true , address=145.79.227.43:4158 , piece_id=74VKW6UWNPLSDKXJ6AG6TTUSDBCW4K4BAGMFIDNIV55AIWRBVB2Q, seg_pos={0 0}, upload=21507]: Long tail cancellation due to optimal threshold Dec 4 17:22:37.332976635 (24h27m13.804187384s) [seg_pos={0 0}, upload=21507]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 4 17:22:37.333114427 (24h27m13.804325166s) [upload=21507]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 4 17:22:37.333121821 (24h27m13.80433255s) [upload=21507]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 4 17:22:37.958438444 (24h27m14.429649173s) [upload=21507]: Done uploading segments... Dec 4 17:22:37.958456208 (24h27m14.429666937s) [upload=21507]: Upload finished. err: <nil> Duration: 5.704392301s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.................X..........................................] duration: 3.525411045s trace: https://jaeger.dev.storj.io/trace/1f3e769edbb82b7a FAIL (2025-12-04 17:35:21 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.525459548s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 5.823260384s trace: https://jaeger.dev.storj.io/trace/400044794bd63bf0 FAIL (2025-12-04 17:35:17 UTC) orbiter warning: read tcp 188.166.225.170:54202->45.38.24.2:443: read: connection reset by peer Duration: 5.823289318s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..................X.........................................] duration: 3.563054497s trace: https://jaeger.dev.storj.io/trace/1f3e769edbb82b7a FAIL (2025-12-04 17:35:05 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.563100424s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 5.690340051s trace: https://jaeger.dev.storj.io/trace/6d15aafc5ce7b199 FAIL (2025-12-04 17:34:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, Piece ID: FBD34B6US36TX6YHN5UEBKOWIVOOP4G7B6SSYKN6Q3J5YNE7OBHA) context deadline exceeded; piecestore: (Node ID: 12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9, Piece ID: ULLE3QB2QSCSYQWZ4OSBDJHC3X7AOHJGRWRA4DUXAMOYBDJRYWRQ) context deadline exceeded; piecestore: (Node ID: 1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, Piece ID: TLP7B6MNA7W35UJQKW5SW6F43YXXMUFXOUKW4DKSOWOPIKLSSVUA) context deadline exceeded; piecestore: (Node ID: 17o881pUMuG7K2dv83k7RkrJCrmM7kgUiWesgam4GEw21KVUn7, Piece ID: QOOXF6G2OGBNJLA64B2YNVVJKAZSP6HM5ZIEI4ZFJKWZUSQAR3XA) context deadline exceeded; piecestore: (Node ID: 1ZrzADk2mKG4R9tjxUKjQeJvb4WmfjLksGFyvstHRfYbBC7XNz, Piece ID: VWS32XVNWH4R5PZDDV4FGP5ONZQZNLMU5YYB7QYS43GDJSDFX2OQ) context deadline exceeded ======LOGS====== Dec 4 17:22:34.320746327 (24h27m10.791957066s) [upload=21507]: Starting upload Dec 4 17:22:34.320860715 (24h27m10.792071454s) [upload=21507]: Done starting upload Dec 4 17:22:34.321208968 (24h27m10.792419707s) [upload=21507]: Uploading segments... Dec 4 17:22:34.321325259 (24h27m10.792536018s) [upload=21507]: Got next segment. Inline: false Dec 4 17:22:34.321394982 (24h27m10.792605711s) [upload=21507]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 17:22:34.321401375 (24h27m10.792612114s) [upload=21507]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 17:22:34.827776318 (24h27m11.298987057s) [seg_pos={0 0}, upload=21507]: Begin upload segment... Dec 4 17:22:34.827838226 (24h27m11.299048966s) [seg_pos={0 0}, upload=21507]: Done begin upload segment. Dec 4 17:22:34.827877361 (24h27m11.29908811s) [noise=true , address=77.90.31.14:4661 , piece_id=D5L5WWE5DGEYKP5M2NM74EGSX5BLEP4DX5OBM2Z4Q2QTCSGA4WXQ, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.827887630 (24h27m11.299098359s) [noise=true , address=77.90.31.21:4878 , piece_id=GHU3SZUB5WAMW37CYCUFJ6HOPMXRNZGIJYDC4DJ3WOHV7YYWCVEA, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.828727861 (24h27m11.2999386s) [noise=true , address=145.79.227.29:4942 , piece_id=QOOXF6G2OGBNJLA64B2YNVVJKAZSP6HM5ZIEI4ZFJKWZUSQAR3XA, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.828821058 (24h27m11.300031807s) [noise=true , address=145.79.227.16:4910 , piece_id=36JF4CMQH5JO66BY5A53QNZQ6RIFTHZ6MD2QLBGHLHML5VBFGK7A, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.829248923 (24h27m11.300459652s) [noise=true , address=145.79.227.43:4157 , piece_id=VWS32XVNWH4R5PZDDV4FGP5ONZQZNLMU5YYB7QYS43GDJSDFX2OQ, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.829618296 (24h27m11.300829035s) [noise=true , address=145.79.227.38:4978 , piece_id=JLYEPL2WLUJIIUHI3FSOWIT5QG3ZQG3NNOZZ3TNMAOEWLZJNN6AA, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.830015262 (24h27m11.301226001s) [noise=true , address=77.90.31.19:4843 , piece_id=36QLYXSVN6L2FDKVX4WS7MRTK54XLB2JUTW2NKHCE7E6KZGTLUKQ, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.830278002 (24h27m11.301488741s) [noise=true , address=145.79.227.21:4475 , piece_id=YZRHRIA7WWLZKDX3BZ2FSUOKJ3MX7D7Q2GB6CHKVQKGXPQCUNC6A, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.830415063 (24h27m11.301625802s) [noise=true , address=145.79.227.8:4329 , piece_id=FBD34B6US36TX6YHN5UEBKOWIVOOP4G7B6SSYKN6Q3J5YNE7OBHA, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.830692902 (24h27m11.301903641s) [noise=true , address=145.79.227.9:4366 , piece_id=GDCC3W66FFB27QD5MRCG2MBJXZBWXRIWG7A4L3XMFTIRNN2NYPUQ, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.830811337 (24h27m11.302022076s) [noise=true , address=77.90.31.10:4520 , piece_id=DGMWUPIGVLYJ4IH2CXCHA3U7FHDJF7APG73XN6TUEWRBB3WZOURQ, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.831349433 (24h27m11.302560172s) [noise=true , address=77.90.31.18:4807 , piece_id=WJ3AOAEE3IG3T3QQRFSKTOJCYVST4263Y5UN6XVOVTSO5ALOVTCQ, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.831805731 (24h27m11.30301647s) [noise=true , address=145.79.227.43:4158 , piece_id=74VKW6UWNPLSDKXJ6AG6TTUSDBCW4K4BAGMFIDNIV55AIWRBVB2Q, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.832172379 (24h27m11.303383118s) [noise=true , address=145.79.227.39:4014 , piece_id=ZLJ7L4OXXXNMJP2NTDU5HNE7NDCFL6JNWLXB6KVZHHFZXF5AT4CA, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.833230595 (24h27m11.304441334s) [noise=true , address=145.79.227.34:4905 , piece_id=ULLE3QB2QSCSYQWZ4OSBDJHC3X7AOHJGRWRA4DUXAMOYBDJRYWRQ, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.833972006 (24h27m11.305182755s) [noise=true , address=145.79.227.10:4401 , piece_id=TLP7B6MNA7W35UJQKW5SW6F43YXXMUFXOUKW4DKSOWOPIKLSSVUA, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:35.930150729 (24h27m12.401361488s) [noise=true , address=77.90.31.19:4843 , piece_id=36QLYXSVN6L2FDKVX4WS7MRTK54XLB2JUTW2NKHCE7E6KZGTLUKQ, seg_pos={0 0}, upload=21507]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: storage node overloaded, request limit: 2000 Dec 4 17:22:36.479057771 (24h27m12.9502685s) [upload=21507]: Next returned nil segment Dec 4 17:22:36.479122765 (24h27m12.950333494s) [upload=21507]: Waiting for error group managing segments... Dec 4 17:22:36.876009453 (24h27m13.347220192s) [noise=true , address=77.90.31.21:4878 , piece_id=GHU3SZUB5WAMW37CYCUFJ6HOPMXRNZGIJYDC4DJ3WOHV7YYWCVEA, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:36.891015988 (24h27m13.362226717s) [noise=true , address=77.90.31.14:4661 , piece_id=D5L5WWE5DGEYKP5M2NM74EGSX5BLEP4DX5OBM2Z4Q2QTCSGA4WXQ, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:36.943731919 (24h27m13.414942658s) [noise=true , address=77.90.31.10:4520 , piece_id=DGMWUPIGVLYJ4IH2CXCHA3U7FHDJF7APG73XN6TUEWRBB3WZOURQ, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:36.981669560 (24h27m13.452880289s) [noise=true , address=145.79.227.16:4910 , piece_id=36JF4CMQH5JO66BY5A53QNZQ6RIFTHZ6MD2QLBGHLHML5VBFGK7A, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:36.981785620 (24h27m13.452996359s) [noise=true , address=145.79.227.21:4475 , piece_id=YZRHRIA7WWLZKDX3BZ2FSUOKJ3MX7D7Q2GB6CHKVQKGXPQCUNC6A, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:36.984955717 (24h27m13.456166456s) [noise=true , address=77.90.31.18:4807 , piece_id=WJ3AOAEE3IG3T3QQRFSKTOJCYVST4263Y5UN6XVOVTSO5ALOVTCQ, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:36.994452571 (24h27m13.4656633s) [noise=true , address=145.79.227.34:4905 , piece_id=ULLE3QB2QSCSYQWZ4OSBDJHC3X7AOHJGRWRA4DUXAMOYBDJRYWRQ, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:37.011055396 (24h27m13.482266125s) [noise=true , address=145.79.227.38:4978 , piece_id=JLYEPL2WLUJIIUHI3FSOWIT5QG3ZQG3NNOZZ3TNMAOEWLZJNN6AA, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:37.021499513 (24h27m13.492710252s) [noise=true , address=145.79.227.8:4329 , piece_id=FBD34B6US36TX6YHN5UEBKOWIVOOP4G7B6SSYKN6Q3J5YNE7OBHA, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:37.043392031 (24h27m13.51460277s) [noise=true , address=145.79.227.39:4014 , piece_id=ZLJ7L4OXXXNMJP2NTDU5HNE7NDCFL6JNWLXB6KVZHHFZXF5AT4CA, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:37.047232213 (24h27m13.518442952s) [noise=true , address=145.79.227.29:4942 , piece_id=QOOXF6G2OGBNJLA64B2YNVVJKAZSP6HM5ZIEI4ZFJKWZUSQAR3XA, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:37.315831267 (24h27m13.787042006s) [noise=true , address=145.79.227.10:4401 , piece_id=TLP7B6MNA7W35UJQKW5SW6F43YXXMUFXOUKW4DKSOWOPIKLSSVUA, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:37.332403453 (24h27m13.803614192s) [noise=true , address=145.79.227.43:4157 , piece_id=VWS32XVNWH4R5PZDDV4FGP5ONZQZNLMU5YYB7QYS43GDJSDFX2OQ, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:37.332432619 (24h27m13.803643358s) [seg_pos={0 0}, upload=21507]: Segment reached optimal threshold of 13 pieces. Dec 4 17:22:37.332650384 (24h27m13.803861123s) [noise=true , address=145.79.227.9:4366 , piece_id=GDCC3W66FFB27QD5MRCG2MBJXZBWXRIWG7A4L3XMFTIRNN2NYPUQ, seg_pos={0 0}, upload=21507]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy): protocol: context canceled Dec 4 17:22:37.332660022 (24h27m13.803870751s) [noise=true , address=145.79.227.9:4366 , piece_id=GDCC3W66FFB27QD5MRCG2MBJXZBWXRIWG7A4L3XMFTIRNN2NYPUQ, seg_pos={0 0}, upload=21507]: Long tail cancellation due to optimal threshold Dec 4 17:22:37.332942179 (24h27m13.804152938s) [noise=true , address=145.79.227.43:4158 , piece_id=74VKW6UWNPLSDKXJ6AG6TTUSDBCW4K4BAGMFIDNIV55AIWRBVB2Q, seg_pos={0 0}, upload=21507]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vJZrSvPvwebw9V2tTbvkPEmLeLjryBUnq9TL6NwdPsHU9xcq2): protocol: expected piece hash; context canceled; EOF Dec 4 17:22:37.332954993 (24h27m13.804165732s) [noise=true , address=145.79.227.43:4158 , piece_id=74VKW6UWNPLSDKXJ6AG6TTUSDBCW4K4BAGMFIDNIV55AIWRBVB2Q, seg_pos={0 0}, upload=21507]: Long tail cancellation due to optimal threshold Dec 4 17:22:37.332976635 (24h27m13.804187384s) [seg_pos={0 0}, upload=21507]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 4 17:22:37.333114427 (24h27m13.804325166s) [upload=21507]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 4 17:22:37.333121821 (24h27m13.80433255s) [upload=21507]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 4 17:22:37.958438444 (24h27m14.429649173s) [upload=21507]: Done uploading segments... Dec 4 17:22:37.958456208 (24h27m14.429666937s) [upload=21507]: Upload finished. err: <nil> Duration: 5.690420704s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...................X........................................] duration: 3.439290584s trace: https://jaeger.dev.storj.io/trace/1f3e769edbb82b7a FAIL (2025-12-04 17:34:23 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.439322145s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 5.815249413s trace: https://jaeger.dev.storj.io/trace/400044794bd63bf0 FAIL (2025-12-04 17:34:19 UTC) orbiter warning: read tcp 188.166.225.170:54202->45.38.24.2:443: read: connection reset by peer Duration: 5.815281424s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 5.263176061s trace: https://jaeger.dev.storj.io/trace/400044794bd63bf0 FAIL (2025-12-04 17:33:47 UTC) orbiter warning: read tcp 188.166.225.170:54202->45.38.24.2:443: read: connection reset by peer Duration: 5.263203653s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 5.667214576s trace: https://jaeger.dev.storj.io/trace/6d15aafc5ce7b199 FAIL (2025-12-04 17:33:41 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, Piece ID: FBD34B6US36TX6YHN5UEBKOWIVOOP4G7B6SSYKN6Q3J5YNE7OBHA) context deadline exceeded; piecestore: (Node ID: 12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9, Piece ID: ULLE3QB2QSCSYQWZ4OSBDJHC3X7AOHJGRWRA4DUXAMOYBDJRYWRQ) context deadline exceeded; piecestore: (Node ID: 1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, Piece ID: TLP7B6MNA7W35UJQKW5SW6F43YXXMUFXOUKW4DKSOWOPIKLSSVUA) context deadline exceeded; piecestore: (Node ID: 17o881pUMuG7K2dv83k7RkrJCrmM7kgUiWesgam4GEw21KVUn7, Piece ID: QOOXF6G2OGBNJLA64B2YNVVJKAZSP6HM5ZIEI4ZFJKWZUSQAR3XA) context deadline exceeded; piecestore: (Node ID: 1ZrzADk2mKG4R9tjxUKjQeJvb4WmfjLksGFyvstHRfYbBC7XNz, Piece ID: VWS32XVNWH4R5PZDDV4FGP5ONZQZNLMU5YYB7QYS43GDJSDFX2OQ) context deadline exceeded ======LOGS====== Dec 4 17:22:34.320746327 (24h27m10.791957066s) [upload=21507]: Starting upload Dec 4 17:22:34.320860715 (24h27m10.792071454s) [upload=21507]: Done starting upload Dec 4 17:22:34.321208968 (24h27m10.792419707s) [upload=21507]: Uploading segments... Dec 4 17:22:34.321325259 (24h27m10.792536018s) [upload=21507]: Got next segment. Inline: false Dec 4 17:22:34.321394982 (24h27m10.792605711s) [upload=21507]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 17:22:34.321401375 (24h27m10.792612114s) [upload=21507]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 17:22:34.827776318 (24h27m11.298987057s) [seg_pos={0 0}, upload=21507]: Begin upload segment... Dec 4 17:22:34.827838226 (24h27m11.299048966s) [seg_pos={0 0}, upload=21507]: Done begin upload segment. Dec 4 17:22:34.827877361 (24h27m11.29908811s) [noise=true , address=77.90.31.14:4661 , piece_id=D5L5WWE5DGEYKP5M2NM74EGSX5BLEP4DX5OBM2Z4Q2QTCSGA4WXQ, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.827887630 (24h27m11.299098359s) [noise=true , address=77.90.31.21:4878 , piece_id=GHU3SZUB5WAMW37CYCUFJ6HOPMXRNZGIJYDC4DJ3WOHV7YYWCVEA, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.828727861 (24h27m11.2999386s) [noise=true , address=145.79.227.29:4942 , piece_id=QOOXF6G2OGBNJLA64B2YNVVJKAZSP6HM5ZIEI4ZFJKWZUSQAR3XA, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.828821058 (24h27m11.300031807s) [noise=true , address=145.79.227.16:4910 , piece_id=36JF4CMQH5JO66BY5A53QNZQ6RIFTHZ6MD2QLBGHLHML5VBFGK7A, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.829248923 (24h27m11.300459652s) [noise=true , address=145.79.227.43:4157 , piece_id=VWS32XVNWH4R5PZDDV4FGP5ONZQZNLMU5YYB7QYS43GDJSDFX2OQ, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.829618296 (24h27m11.300829035s) [noise=true , address=145.79.227.38:4978 , piece_id=JLYEPL2WLUJIIUHI3FSOWIT5QG3ZQG3NNOZZ3TNMAOEWLZJNN6AA, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.830015262 (24h27m11.301226001s) [noise=true , address=77.90.31.19:4843 , piece_id=36QLYXSVN6L2FDKVX4WS7MRTK54XLB2JUTW2NKHCE7E6KZGTLUKQ, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.830278002 (24h27m11.301488741s) [noise=true , address=145.79.227.21:4475 , piece_id=YZRHRIA7WWLZKDX3BZ2FSUOKJ3MX7D7Q2GB6CHKVQKGXPQCUNC6A, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.830415063 (24h27m11.301625802s) [noise=true , address=145.79.227.8:4329 , piece_id=FBD34B6US36TX6YHN5UEBKOWIVOOP4G7B6SSYKN6Q3J5YNE7OBHA, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.830692902 (24h27m11.301903641s) [noise=true , address=145.79.227.9:4366 , piece_id=GDCC3W66FFB27QD5MRCG2MBJXZBWXRIWG7A4L3XMFTIRNN2NYPUQ, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.830811337 (24h27m11.302022076s) [noise=true , address=77.90.31.10:4520 , piece_id=DGMWUPIGVLYJ4IH2CXCHA3U7FHDJF7APG73XN6TUEWRBB3WZOURQ, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.831349433 (24h27m11.302560172s) [noise=true , address=77.90.31.18:4807 , piece_id=WJ3AOAEE3IG3T3QQRFSKTOJCYVST4263Y5UN6XVOVTSO5ALOVTCQ, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.831805731 (24h27m11.30301647s) [noise=true , address=145.79.227.43:4158 , piece_id=74VKW6UWNPLSDKXJ6AG6TTUSDBCW4K4BAGMFIDNIV55AIWRBVB2Q, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.832172379 (24h27m11.303383118s) [noise=true , address=145.79.227.39:4014 , piece_id=ZLJ7L4OXXXNMJP2NTDU5HNE7NDCFL6JNWLXB6KVZHHFZXF5AT4CA, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.833230595 (24h27m11.304441334s) [noise=true , address=145.79.227.34:4905 , piece_id=ULLE3QB2QSCSYQWZ4OSBDJHC3X7AOHJGRWRA4DUXAMOYBDJRYWRQ, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.833972006 (24h27m11.305182755s) [noise=true , address=145.79.227.10:4401 , piece_id=TLP7B6MNA7W35UJQKW5SW6F43YXXMUFXOUKW4DKSOWOPIKLSSVUA, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:35.930150729 (24h27m12.401361488s) [noise=true , address=77.90.31.19:4843 , piece_id=36QLYXSVN6L2FDKVX4WS7MRTK54XLB2JUTW2NKHCE7E6KZGTLUKQ, seg_pos={0 0}, upload=21507]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: storage node overloaded, request limit: 2000 Dec 4 17:22:36.479057771 (24h27m12.9502685s) [upload=21507]: Next returned nil segment Dec 4 17:22:36.479122765 (24h27m12.950333494s) [upload=21507]: Waiting for error group managing segments... Dec 4 17:22:36.876009453 (24h27m13.347220192s) [noise=true , address=77.90.31.21:4878 , piece_id=GHU3SZUB5WAMW37CYCUFJ6HOPMXRNZGIJYDC4DJ3WOHV7YYWCVEA, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:36.891015988 (24h27m13.362226717s) [noise=true , address=77.90.31.14:4661 , piece_id=D5L5WWE5DGEYKP5M2NM74EGSX5BLEP4DX5OBM2Z4Q2QTCSGA4WXQ, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:36.943731919 (24h27m13.414942658s) [noise=true , address=77.90.31.10:4520 , piece_id=DGMWUPIGVLYJ4IH2CXCHA3U7FHDJF7APG73XN6TUEWRBB3WZOURQ, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:36.981669560 (24h27m13.452880289s) [noise=true , address=145.79.227.16:4910 , piece_id=36JF4CMQH5JO66BY5A53QNZQ6RIFTHZ6MD2QLBGHLHML5VBFGK7A, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:36.981785620 (24h27m13.452996359s) [noise=true , address=145.79.227.21:4475 , piece_id=YZRHRIA7WWLZKDX3BZ2FSUOKJ3MX7D7Q2GB6CHKVQKGXPQCUNC6A, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:36.984955717 (24h27m13.456166456s) [noise=true , address=77.90.31.18:4807 , piece_id=WJ3AOAEE3IG3T3QQRFSKTOJCYVST4263Y5UN6XVOVTSO5ALOVTCQ, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:36.994452571 (24h27m13.4656633s) [noise=true , address=145.79.227.34:4905 , piece_id=ULLE3QB2QSCSYQWZ4OSBDJHC3X7AOHJGRWRA4DUXAMOYBDJRYWRQ, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:37.011055396 (24h27m13.482266125s) [noise=true , address=145.79.227.38:4978 , piece_id=JLYEPL2WLUJIIUHI3FSOWIT5QG3ZQG3NNOZZ3TNMAOEWLZJNN6AA, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:37.021499513 (24h27m13.492710252s) [noise=true , address=145.79.227.8:4329 , piece_id=FBD34B6US36TX6YHN5UEBKOWIVOOP4G7B6SSYKN6Q3J5YNE7OBHA, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:37.043392031 (24h27m13.51460277s) [noise=true , address=145.79.227.39:4014 , piece_id=ZLJ7L4OXXXNMJP2NTDU5HNE7NDCFL6JNWLXB6KVZHHFZXF5AT4CA, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:37.047232213 (24h27m13.518442952s) [noise=true , address=145.79.227.29:4942 , piece_id=QOOXF6G2OGBNJLA64B2YNVVJKAZSP6HM5ZIEI4ZFJKWZUSQAR3XA, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:37.315831267 (24h27m13.787042006s) [noise=true , address=145.79.227.10:4401 , piece_id=TLP7B6MNA7W35UJQKW5SW6F43YXXMUFXOUKW4DKSOWOPIKLSSVUA, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:37.332403453 (24h27m13.803614192s) [noise=true , address=145.79.227.43:4157 , piece_id=VWS32XVNWH4R5PZDDV4FGP5ONZQZNLMU5YYB7QYS43GDJSDFX2OQ, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:37.332432619 (24h27m13.803643358s) [seg_pos={0 0}, upload=21507]: Segment reached optimal threshold of 13 pieces. Dec 4 17:22:37.332650384 (24h27m13.803861123s) [noise=true , address=145.79.227.9:4366 , piece_id=GDCC3W66FFB27QD5MRCG2MBJXZBWXRIWG7A4L3XMFTIRNN2NYPUQ, seg_pos={0 0}, upload=21507]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy): protocol: context canceled Dec 4 17:22:37.332660022 (24h27m13.803870751s) [noise=true , address=145.79.227.9:4366 , piece_id=GDCC3W66FFB27QD5MRCG2MBJXZBWXRIWG7A4L3XMFTIRNN2NYPUQ, seg_pos={0 0}, upload=21507]: Long tail cancellation due to optimal threshold Dec 4 17:22:37.332942179 (24h27m13.804152938s) [noise=true , address=145.79.227.43:4158 , piece_id=74VKW6UWNPLSDKXJ6AG6TTUSDBCW4K4BAGMFIDNIV55AIWRBVB2Q, seg_pos={0 0}, upload=21507]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vJZrSvPvwebw9V2tTbvkPEmLeLjryBUnq9TL6NwdPsHU9xcq2): protocol: expected piece hash; context canceled; EOF Dec 4 17:22:37.332954993 (24h27m13.804165732s) [noise=true , address=145.79.227.43:4158 , piece_id=74VKW6UWNPLSDKXJ6AG6TTUSDBCW4K4BAGMFIDNIV55AIWRBVB2Q, seg_pos={0 0}, upload=21507]: Long tail cancellation due to optimal threshold Dec 4 17:22:37.332976635 (24h27m13.804187384s) [seg_pos={0 0}, upload=21507]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 4 17:22:37.333114427 (24h27m13.804325166s) [upload=21507]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 4 17:22:37.333121821 (24h27m13.80433255s) [upload=21507]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 4 17:22:37.958438444 (24h27m14.429649173s) [upload=21507]: Done uploading segments... Dec 4 17:22:37.958456208 (24h27m14.429666937s) [upload=21507]: Upload finished. err: <nil> Duration: 5.667234253s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [....................X.......................................] duration: 3.384320977s trace: https://jaeger.dev.storj.io/trace/1f3e769edbb82b7a FAIL (2025-12-04 17:33:29 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.384360761s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 5.799514723s trace: https://jaeger.dev.storj.io/trace/6d15aafc5ce7b199 FAIL (2025-12-04 17:33:20 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, Piece ID: FBD34B6US36TX6YHN5UEBKOWIVOOP4G7B6SSYKN6Q3J5YNE7OBHA) context deadline exceeded; piecestore: (Node ID: 12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9, Piece ID: ULLE3QB2QSCSYQWZ4OSBDJHC3X7AOHJGRWRA4DUXAMOYBDJRYWRQ) context deadline exceeded; piecestore: (Node ID: 1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, Piece ID: TLP7B6MNA7W35UJQKW5SW6F43YXXMUFXOUKW4DKSOWOPIKLSSVUA) context deadline exceeded; piecestore: (Node ID: 17o881pUMuG7K2dv83k7RkrJCrmM7kgUiWesgam4GEw21KVUn7, Piece ID: QOOXF6G2OGBNJLA64B2YNVVJKAZSP6HM5ZIEI4ZFJKWZUSQAR3XA) context deadline exceeded; piecestore: (Node ID: 1ZrzADk2mKG4R9tjxUKjQeJvb4WmfjLksGFyvstHRfYbBC7XNz, Piece ID: VWS32XVNWH4R5PZDDV4FGP5ONZQZNLMU5YYB7QYS43GDJSDFX2OQ) context deadline exceeded ======LOGS====== Dec 4 17:22:34.320746327 (24h27m10.791957066s) [upload=21507]: Starting upload Dec 4 17:22:34.320860715 (24h27m10.792071454s) [upload=21507]: Done starting upload Dec 4 17:22:34.321208968 (24h27m10.792419707s) [upload=21507]: Uploading segments... Dec 4 17:22:34.321325259 (24h27m10.792536018s) [upload=21507]: Got next segment. Inline: false Dec 4 17:22:34.321394982 (24h27m10.792605711s) [upload=21507]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 17:22:34.321401375 (24h27m10.792612114s) [upload=21507]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 17:22:34.827776318 (24h27m11.298987057s) [seg_pos={0 0}, upload=21507]: Begin upload segment... Dec 4 17:22:34.827838226 (24h27m11.299048966s) [seg_pos={0 0}, upload=21507]: Done begin upload segment. Dec 4 17:22:34.827877361 (24h27m11.29908811s) [noise=true , address=77.90.31.14:4661 , piece_id=D5L5WWE5DGEYKP5M2NM74EGSX5BLEP4DX5OBM2Z4Q2QTCSGA4WXQ, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.827887630 (24h27m11.299098359s) [noise=true , address=77.90.31.21:4878 , piece_id=GHU3SZUB5WAMW37CYCUFJ6HOPMXRNZGIJYDC4DJ3WOHV7YYWCVEA, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.828727861 (24h27m11.2999386s) [noise=true , address=145.79.227.29:4942 , piece_id=QOOXF6G2OGBNJLA64B2YNVVJKAZSP6HM5ZIEI4ZFJKWZUSQAR3XA, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.828821058 (24h27m11.300031807s) [noise=true , address=145.79.227.16:4910 , piece_id=36JF4CMQH5JO66BY5A53QNZQ6RIFTHZ6MD2QLBGHLHML5VBFGK7A, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.829248923 (24h27m11.300459652s) [noise=true , address=145.79.227.43:4157 , piece_id=VWS32XVNWH4R5PZDDV4FGP5ONZQZNLMU5YYB7QYS43GDJSDFX2OQ, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.829618296 (24h27m11.300829035s) [noise=true , address=145.79.227.38:4978 , piece_id=JLYEPL2WLUJIIUHI3FSOWIT5QG3ZQG3NNOZZ3TNMAOEWLZJNN6AA, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.830015262 (24h27m11.301226001s) [noise=true , address=77.90.31.19:4843 , piece_id=36QLYXSVN6L2FDKVX4WS7MRTK54XLB2JUTW2NKHCE7E6KZGTLUKQ, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.830278002 (24h27m11.301488741s) [noise=true , address=145.79.227.21:4475 , piece_id=YZRHRIA7WWLZKDX3BZ2FSUOKJ3MX7D7Q2GB6CHKVQKGXPQCUNC6A, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.830415063 (24h27m11.301625802s) [noise=true , address=145.79.227.8:4329 , piece_id=FBD34B6US36TX6YHN5UEBKOWIVOOP4G7B6SSYKN6Q3J5YNE7OBHA, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.830692902 (24h27m11.301903641s) [noise=true , address=145.79.227.9:4366 , piece_id=GDCC3W66FFB27QD5MRCG2MBJXZBWXRIWG7A4L3XMFTIRNN2NYPUQ, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.830811337 (24h27m11.302022076s) [noise=true , address=77.90.31.10:4520 , piece_id=DGMWUPIGVLYJ4IH2CXCHA3U7FHDJF7APG73XN6TUEWRBB3WZOURQ, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.831349433 (24h27m11.302560172s) [noise=true , address=77.90.31.18:4807 , piece_id=WJ3AOAEE3IG3T3QQRFSKTOJCYVST4263Y5UN6XVOVTSO5ALOVTCQ, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.831805731 (24h27m11.30301647s) [noise=true , address=145.79.227.43:4158 , piece_id=74VKW6UWNPLSDKXJ6AG6TTUSDBCW4K4BAGMFIDNIV55AIWRBVB2Q, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.832172379 (24h27m11.303383118s) [noise=true , address=145.79.227.39:4014 , piece_id=ZLJ7L4OXXXNMJP2NTDU5HNE7NDCFL6JNWLXB6KVZHHFZXF5AT4CA, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.833230595 (24h27m11.304441334s) [noise=true , address=145.79.227.34:4905 , piece_id=ULLE3QB2QSCSYQWZ4OSBDJHC3X7AOHJGRWRA4DUXAMOYBDJRYWRQ, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.833972006 (24h27m11.305182755s) [noise=true , address=145.79.227.10:4401 , piece_id=TLP7B6MNA7W35UJQKW5SW6F43YXXMUFXOUKW4DKSOWOPIKLSSVUA, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:35.930150729 (24h27m12.401361488s) [noise=true , address=77.90.31.19:4843 , piece_id=36QLYXSVN6L2FDKVX4WS7MRTK54XLB2JUTW2NKHCE7E6KZGTLUKQ, seg_pos={0 0}, upload=21507]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: storage node overloaded, request limit: 2000 Dec 4 17:22:36.479057771 (24h27m12.9502685s) [upload=21507]: Next returned nil segment Dec 4 17:22:36.479122765 (24h27m12.950333494s) [upload=21507]: Waiting for error group managing segments... Dec 4 17:22:36.876009453 (24h27m13.347220192s) [noise=true , address=77.90.31.21:4878 , piece_id=GHU3SZUB5WAMW37CYCUFJ6HOPMXRNZGIJYDC4DJ3WOHV7YYWCVEA, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:36.891015988 (24h27m13.362226717s) [noise=true , address=77.90.31.14:4661 , piece_id=D5L5WWE5DGEYKP5M2NM74EGSX5BLEP4DX5OBM2Z4Q2QTCSGA4WXQ, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:36.943731919 (24h27m13.414942658s) [noise=true , address=77.90.31.10:4520 , piece_id=DGMWUPIGVLYJ4IH2CXCHA3U7FHDJF7APG73XN6TUEWRBB3WZOURQ, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:36.981669560 (24h27m13.452880289s) [noise=true , address=145.79.227.16:4910 , piece_id=36JF4CMQH5JO66BY5A53QNZQ6RIFTHZ6MD2QLBGHLHML5VBFGK7A, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:36.981785620 (24h27m13.452996359s) [noise=true , address=145.79.227.21:4475 , piece_id=YZRHRIA7WWLZKDX3BZ2FSUOKJ3MX7D7Q2GB6CHKVQKGXPQCUNC6A, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:36.984955717 (24h27m13.456166456s) [noise=true , address=77.90.31.18:4807 , piece_id=WJ3AOAEE3IG3T3QQRFSKTOJCYVST4263Y5UN6XVOVTSO5ALOVTCQ, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:36.994452571 (24h27m13.4656633s) [noise=true , address=145.79.227.34:4905 , piece_id=ULLE3QB2QSCSYQWZ4OSBDJHC3X7AOHJGRWRA4DUXAMOYBDJRYWRQ, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:37.011055396 (24h27m13.482266125s) [noise=true , address=145.79.227.38:4978 , piece_id=JLYEPL2WLUJIIUHI3FSOWIT5QG3ZQG3NNOZZ3TNMAOEWLZJNN6AA, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:37.021499513 (24h27m13.492710252s) [noise=true , address=145.79.227.8:4329 , piece_id=FBD34B6US36TX6YHN5UEBKOWIVOOP4G7B6SSYKN6Q3J5YNE7OBHA, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:37.043392031 (24h27m13.51460277s) [noise=true , address=145.79.227.39:4014 , piece_id=ZLJ7L4OXXXNMJP2NTDU5HNE7NDCFL6JNWLXB6KVZHHFZXF5AT4CA, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:37.047232213 (24h27m13.518442952s) [noise=true , address=145.79.227.29:4942 , piece_id=QOOXF6G2OGBNJLA64B2YNVVJKAZSP6HM5ZIEI4ZFJKWZUSQAR3XA, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:37.315831267 (24h27m13.787042006s) [noise=true , address=145.79.227.10:4401 , piece_id=TLP7B6MNA7W35UJQKW5SW6F43YXXMUFXOUKW4DKSOWOPIKLSSVUA, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:37.332403453 (24h27m13.803614192s) [noise=true , address=145.79.227.43:4157 , piece_id=VWS32XVNWH4R5PZDDV4FGP5ONZQZNLMU5YYB7QYS43GDJSDFX2OQ, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:37.332432619 (24h27m13.803643358s) [seg_pos={0 0}, upload=21507]: Segment reached optimal threshold of 13 pieces. Dec 4 17:22:37.332650384 (24h27m13.803861123s) [noise=true , address=145.79.227.9:4366 , piece_id=GDCC3W66FFB27QD5MRCG2MBJXZBWXRIWG7A4L3XMFTIRNN2NYPUQ, seg_pos={0 0}, upload=21507]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy): protocol: context canceled Dec 4 17:22:37.332660022 (24h27m13.803870751s) [noise=true , address=145.79.227.9:4366 , piece_id=GDCC3W66FFB27QD5MRCG2MBJXZBWXRIWG7A4L3XMFTIRNN2NYPUQ, seg_pos={0 0}, upload=21507]: Long tail cancellation due to optimal threshold Dec 4 17:22:37.332942179 (24h27m13.804152938s) [noise=true , address=145.79.227.43:4158 , piece_id=74VKW6UWNPLSDKXJ6AG6TTUSDBCW4K4BAGMFIDNIV55AIWRBVB2Q, seg_pos={0 0}, upload=21507]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vJZrSvPvwebw9V2tTbvkPEmLeLjryBUnq9TL6NwdPsHU9xcq2): protocol: expected piece hash; context canceled; EOF Dec 4 17:22:37.332954993 (24h27m13.804165732s) [noise=true , address=145.79.227.43:4158 , piece_id=74VKW6UWNPLSDKXJ6AG6TTUSDBCW4K4BAGMFIDNIV55AIWRBVB2Q, seg_pos={0 0}, upload=21507]: Long tail cancellation due to optimal threshold Dec 4 17:22:37.332976635 (24h27m13.804187384s) [seg_pos={0 0}, upload=21507]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 4 17:22:37.333114427 (24h27m13.804325166s) [upload=21507]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 4 17:22:37.333121821 (24h27m13.80433255s) [upload=21507]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 4 17:22:37.958438444 (24h27m14.429649173s) [upload=21507]: Done uploading segments... Dec 4 17:22:37.958456208 (24h27m14.429666937s) [upload=21507]: Upload finished. err: <nil> Duration: 5.799541113s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 5.731577555s trace: https://jaeger.dev.storj.io/trace/6d15aafc5ce7b199 FAIL (2025-12-04 17:32:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, Piece ID: FBD34B6US36TX6YHN5UEBKOWIVOOP4G7B6SSYKN6Q3J5YNE7OBHA) context deadline exceeded; piecestore: (Node ID: 12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9, Piece ID: ULLE3QB2QSCSYQWZ4OSBDJHC3X7AOHJGRWRA4DUXAMOYBDJRYWRQ) context deadline exceeded; piecestore: (Node ID: 1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, Piece ID: TLP7B6MNA7W35UJQKW5SW6F43YXXMUFXOUKW4DKSOWOPIKLSSVUA) context deadline exceeded; piecestore: (Node ID: 17o881pUMuG7K2dv83k7RkrJCrmM7kgUiWesgam4GEw21KVUn7, Piece ID: QOOXF6G2OGBNJLA64B2YNVVJKAZSP6HM5ZIEI4ZFJKWZUSQAR3XA) context deadline exceeded; piecestore: (Node ID: 1ZrzADk2mKG4R9tjxUKjQeJvb4WmfjLksGFyvstHRfYbBC7XNz, Piece ID: VWS32XVNWH4R5PZDDV4FGP5ONZQZNLMU5YYB7QYS43GDJSDFX2OQ) context deadline exceeded ======LOGS====== Dec 4 17:22:34.320746327 (24h27m10.791957066s) [upload=21507]: Starting upload Dec 4 17:22:34.320860715 (24h27m10.792071454s) [upload=21507]: Done starting upload Dec 4 17:22:34.321208968 (24h27m10.792419707s) [upload=21507]: Uploading segments... Dec 4 17:22:34.321325259 (24h27m10.792536018s) [upload=21507]: Got next segment. Inline: false Dec 4 17:22:34.321394982 (24h27m10.792605711s) [upload=21507]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 17:22:34.321401375 (24h27m10.792612114s) [upload=21507]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 17:22:34.827776318 (24h27m11.298987057s) [seg_pos={0 0}, upload=21507]: Begin upload segment... Dec 4 17:22:34.827838226 (24h27m11.299048966s) [seg_pos={0 0}, upload=21507]: Done begin upload segment. Dec 4 17:22:34.827877361 (24h27m11.29908811s) [noise=true , address=77.90.31.14:4661 , piece_id=D5L5WWE5DGEYKP5M2NM74EGSX5BLEP4DX5OBM2Z4Q2QTCSGA4WXQ, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.827887630 (24h27m11.299098359s) [noise=true , address=77.90.31.21:4878 , piece_id=GHU3SZUB5WAMW37CYCUFJ6HOPMXRNZGIJYDC4DJ3WOHV7YYWCVEA, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.828727861 (24h27m11.2999386s) [noise=true , address=145.79.227.29:4942 , piece_id=QOOXF6G2OGBNJLA64B2YNVVJKAZSP6HM5ZIEI4ZFJKWZUSQAR3XA, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.828821058 (24h27m11.300031807s) [noise=true , address=145.79.227.16:4910 , piece_id=36JF4CMQH5JO66BY5A53QNZQ6RIFTHZ6MD2QLBGHLHML5VBFGK7A, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.829248923 (24h27m11.300459652s) [noise=true , address=145.79.227.43:4157 , piece_id=VWS32XVNWH4R5PZDDV4FGP5ONZQZNLMU5YYB7QYS43GDJSDFX2OQ, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.829618296 (24h27m11.300829035s) [noise=true , address=145.79.227.38:4978 , piece_id=JLYEPL2WLUJIIUHI3FSOWIT5QG3ZQG3NNOZZ3TNMAOEWLZJNN6AA, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.830015262 (24h27m11.301226001s) [noise=true , address=77.90.31.19:4843 , piece_id=36QLYXSVN6L2FDKVX4WS7MRTK54XLB2JUTW2NKHCE7E6KZGTLUKQ, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.830278002 (24h27m11.301488741s) [noise=true , address=145.79.227.21:4475 , piece_id=YZRHRIA7WWLZKDX3BZ2FSUOKJ3MX7D7Q2GB6CHKVQKGXPQCUNC6A, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.830415063 (24h27m11.301625802s) [noise=true , address=145.79.227.8:4329 , piece_id=FBD34B6US36TX6YHN5UEBKOWIVOOP4G7B6SSYKN6Q3J5YNE7OBHA, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.830692902 (24h27m11.301903641s) [noise=true , address=145.79.227.9:4366 , piece_id=GDCC3W66FFB27QD5MRCG2MBJXZBWXRIWG7A4L3XMFTIRNN2NYPUQ, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.830811337 (24h27m11.302022076s) [noise=true , address=77.90.31.10:4520 , piece_id=DGMWUPIGVLYJ4IH2CXCHA3U7FHDJF7APG73XN6TUEWRBB3WZOURQ, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.831349433 (24h27m11.302560172s) [noise=true , address=77.90.31.18:4807 , piece_id=WJ3AOAEE3IG3T3QQRFSKTOJCYVST4263Y5UN6XVOVTSO5ALOVTCQ, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.831805731 (24h27m11.30301647s) [noise=true , address=145.79.227.43:4158 , piece_id=74VKW6UWNPLSDKXJ6AG6TTUSDBCW4K4BAGMFIDNIV55AIWRBVB2Q, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.832172379 (24h27m11.303383118s) [noise=true , address=145.79.227.39:4014 , piece_id=ZLJ7L4OXXXNMJP2NTDU5HNE7NDCFL6JNWLXB6KVZHHFZXF5AT4CA, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.833230595 (24h27m11.304441334s) [noise=true , address=145.79.227.34:4905 , piece_id=ULLE3QB2QSCSYQWZ4OSBDJHC3X7AOHJGRWRA4DUXAMOYBDJRYWRQ, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.833972006 (24h27m11.305182755s) [noise=true , address=145.79.227.10:4401 , piece_id=TLP7B6MNA7W35UJQKW5SW6F43YXXMUFXOUKW4DKSOWOPIKLSSVUA, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:35.930150729 (24h27m12.401361488s) [noise=true , address=77.90.31.19:4843 , piece_id=36QLYXSVN6L2FDKVX4WS7MRTK54XLB2JUTW2NKHCE7E6KZGTLUKQ, seg_pos={0 0}, upload=21507]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: storage node overloaded, request limit: 2000 Dec 4 17:22:36.479057771 (24h27m12.9502685s) [upload=21507]: Next returned nil segment Dec 4 17:22:36.479122765 (24h27m12.950333494s) [upload=21507]: Waiting for error group managing segments... Dec 4 17:22:36.876009453 (24h27m13.347220192s) [noise=true , address=77.90.31.21:4878 , piece_id=GHU3SZUB5WAMW37CYCUFJ6HOPMXRNZGIJYDC4DJ3WOHV7YYWCVEA, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:36.891015988 (24h27m13.362226717s) [noise=true , address=77.90.31.14:4661 , piece_id=D5L5WWE5DGEYKP5M2NM74EGSX5BLEP4DX5OBM2Z4Q2QTCSGA4WXQ, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:36.943731919 (24h27m13.414942658s) [noise=true , address=77.90.31.10:4520 , piece_id=DGMWUPIGVLYJ4IH2CXCHA3U7FHDJF7APG73XN6TUEWRBB3WZOURQ, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:36.981669560 (24h27m13.452880289s) [noise=true , address=145.79.227.16:4910 , piece_id=36JF4CMQH5JO66BY5A53QNZQ6RIFTHZ6MD2QLBGHLHML5VBFGK7A, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:36.981785620 (24h27m13.452996359s) [noise=true , address=145.79.227.21:4475 , piece_id=YZRHRIA7WWLZKDX3BZ2FSUOKJ3MX7D7Q2GB6CHKVQKGXPQCUNC6A, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:36.984955717 (24h27m13.456166456s) [noise=true , address=77.90.31.18:4807 , piece_id=WJ3AOAEE3IG3T3QQRFSKTOJCYVST4263Y5UN6XVOVTSO5ALOVTCQ, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:36.994452571 (24h27m13.4656633s) [noise=true , address=145.79.227.34:4905 , piece_id=ULLE3QB2QSCSYQWZ4OSBDJHC3X7AOHJGRWRA4DUXAMOYBDJRYWRQ, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:37.011055396 (24h27m13.482266125s) [noise=true , address=145.79.227.38:4978 , piece_id=JLYEPL2WLUJIIUHI3FSOWIT5QG3ZQG3NNOZZ3TNMAOEWLZJNN6AA, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:37.021499513 (24h27m13.492710252s) [noise=true , address=145.79.227.8:4329 , piece_id=FBD34B6US36TX6YHN5UEBKOWIVOOP4G7B6SSYKN6Q3J5YNE7OBHA, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:37.043392031 (24h27m13.51460277s) [noise=true , address=145.79.227.39:4014 , piece_id=ZLJ7L4OXXXNMJP2NTDU5HNE7NDCFL6JNWLXB6KVZHHFZXF5AT4CA, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:37.047232213 (24h27m13.518442952s) [noise=true , address=145.79.227.29:4942 , piece_id=QOOXF6G2OGBNJLA64B2YNVVJKAZSP6HM5ZIEI4ZFJKWZUSQAR3XA, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:37.315831267 (24h27m13.787042006s) [noise=true , address=145.79.227.10:4401 , piece_id=TLP7B6MNA7W35UJQKW5SW6F43YXXMUFXOUKW4DKSOWOPIKLSSVUA, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:37.332403453 (24h27m13.803614192s) [noise=true , address=145.79.227.43:4157 , piece_id=VWS32XVNWH4R5PZDDV4FGP5ONZQZNLMU5YYB7QYS43GDJSDFX2OQ, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:37.332432619 (24h27m13.803643358s) [seg_pos={0 0}, upload=21507]: Segment reached optimal threshold of 13 pieces. Dec 4 17:22:37.332650384 (24h27m13.803861123s) [noise=true , address=145.79.227.9:4366 , piece_id=GDCC3W66FFB27QD5MRCG2MBJXZBWXRIWG7A4L3XMFTIRNN2NYPUQ, seg_pos={0 0}, upload=21507]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy): protocol: context canceled Dec 4 17:22:37.332660022 (24h27m13.803870751s) [noise=true , address=145.79.227.9:4366 , piece_id=GDCC3W66FFB27QD5MRCG2MBJXZBWXRIWG7A4L3XMFTIRNN2NYPUQ, seg_pos={0 0}, upload=21507]: Long tail cancellation due to optimal threshold Dec 4 17:22:37.332942179 (24h27m13.804152938s) [noise=true , address=145.79.227.43:4158 , piece_id=74VKW6UWNPLSDKXJ6AG6TTUSDBCW4K4BAGMFIDNIV55AIWRBVB2Q, seg_pos={0 0}, upload=21507]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vJZrSvPvwebw9V2tTbvkPEmLeLjryBUnq9TL6NwdPsHU9xcq2): protocol: expected piece hash; context canceled; EOF Dec 4 17:22:37.332954993 (24h27m13.804165732s) [noise=true , address=145.79.227.43:4158 , piece_id=74VKW6UWNPLSDKXJ6AG6TTUSDBCW4K4BAGMFIDNIV55AIWRBVB2Q, seg_pos={0 0}, upload=21507]: Long tail cancellation due to optimal threshold Dec 4 17:22:37.332976635 (24h27m13.804187384s) [seg_pos={0 0}, upload=21507]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 4 17:22:37.333114427 (24h27m13.804325166s) [upload=21507]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 4 17:22:37.333121821 (24h27m13.80433255s) [upload=21507]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 4 17:22:37.958438444 (24h27m14.429649173s) [upload=21507]: Done uploading segments... Dec 4 17:22:37.958456208 (24h27m14.429666937s) [upload=21507]: Upload finished. err: <nil> Duration: 5.731604607s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.....................X......................................] duration: 3.434164104s trace: https://jaeger.dev.storj.io/trace/1f3e769edbb82b7a FAIL (2025-12-04 17:32:21 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.43419406s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 5.902610938s trace: https://jaeger.dev.storj.io/trace/6d15aafc5ce7b199 FAIL (2025-12-04 17:31:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, Piece ID: FBD34B6US36TX6YHN5UEBKOWIVOOP4G7B6SSYKN6Q3J5YNE7OBHA) context deadline exceeded; piecestore: (Node ID: 12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9, Piece ID: ULLE3QB2QSCSYQWZ4OSBDJHC3X7AOHJGRWRA4DUXAMOYBDJRYWRQ) context deadline exceeded; piecestore: (Node ID: 1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, Piece ID: TLP7B6MNA7W35UJQKW5SW6F43YXXMUFXOUKW4DKSOWOPIKLSSVUA) context deadline exceeded; piecestore: (Node ID: 17o881pUMuG7K2dv83k7RkrJCrmM7kgUiWesgam4GEw21KVUn7, Piece ID: QOOXF6G2OGBNJLA64B2YNVVJKAZSP6HM5ZIEI4ZFJKWZUSQAR3XA) context deadline exceeded; piecestore: (Node ID: 1ZrzADk2mKG4R9tjxUKjQeJvb4WmfjLksGFyvstHRfYbBC7XNz, Piece ID: VWS32XVNWH4R5PZDDV4FGP5ONZQZNLMU5YYB7QYS43GDJSDFX2OQ) context deadline exceeded ======LOGS====== Dec 4 17:22:34.320746327 (24h27m10.791957066s) [upload=21507]: Starting upload Dec 4 17:22:34.320860715 (24h27m10.792071454s) [upload=21507]: Done starting upload Dec 4 17:22:34.321208968 (24h27m10.792419707s) [upload=21507]: Uploading segments... Dec 4 17:22:34.321325259 (24h27m10.792536018s) [upload=21507]: Got next segment. Inline: false Dec 4 17:22:34.321394982 (24h27m10.792605711s) [upload=21507]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 17:22:34.321401375 (24h27m10.792612114s) [upload=21507]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 17:22:34.827776318 (24h27m11.298987057s) [seg_pos={0 0}, upload=21507]: Begin upload segment... Dec 4 17:22:34.827838226 (24h27m11.299048966s) [seg_pos={0 0}, upload=21507]: Done begin upload segment. Dec 4 17:22:34.827877361 (24h27m11.29908811s) [noise=true , address=77.90.31.14:4661 , piece_id=D5L5WWE5DGEYKP5M2NM74EGSX5BLEP4DX5OBM2Z4Q2QTCSGA4WXQ, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.827887630 (24h27m11.299098359s) [noise=true , address=77.90.31.21:4878 , piece_id=GHU3SZUB5WAMW37CYCUFJ6HOPMXRNZGIJYDC4DJ3WOHV7YYWCVEA, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.828727861 (24h27m11.2999386s) [noise=true , address=145.79.227.29:4942 , piece_id=QOOXF6G2OGBNJLA64B2YNVVJKAZSP6HM5ZIEI4ZFJKWZUSQAR3XA, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.828821058 (24h27m11.300031807s) [noise=true , address=145.79.227.16:4910 , piece_id=36JF4CMQH5JO66BY5A53QNZQ6RIFTHZ6MD2QLBGHLHML5VBFGK7A, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.829248923 (24h27m11.300459652s) [noise=true , address=145.79.227.43:4157 , piece_id=VWS32XVNWH4R5PZDDV4FGP5ONZQZNLMU5YYB7QYS43GDJSDFX2OQ, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.829618296 (24h27m11.300829035s) [noise=true , address=145.79.227.38:4978 , piece_id=JLYEPL2WLUJIIUHI3FSOWIT5QG3ZQG3NNOZZ3TNMAOEWLZJNN6AA, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.830015262 (24h27m11.301226001s) [noise=true , address=77.90.31.19:4843 , piece_id=36QLYXSVN6L2FDKVX4WS7MRTK54XLB2JUTW2NKHCE7E6KZGTLUKQ, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.830278002 (24h27m11.301488741s) [noise=true , address=145.79.227.21:4475 , piece_id=YZRHRIA7WWLZKDX3BZ2FSUOKJ3MX7D7Q2GB6CHKVQKGXPQCUNC6A, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.830415063 (24h27m11.301625802s) [noise=true , address=145.79.227.8:4329 , piece_id=FBD34B6US36TX6YHN5UEBKOWIVOOP4G7B6SSYKN6Q3J5YNE7OBHA, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.830692902 (24h27m11.301903641s) [noise=true , address=145.79.227.9:4366 , piece_id=GDCC3W66FFB27QD5MRCG2MBJXZBWXRIWG7A4L3XMFTIRNN2NYPUQ, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.830811337 (24h27m11.302022076s) [noise=true , address=77.90.31.10:4520 , piece_id=DGMWUPIGVLYJ4IH2CXCHA3U7FHDJF7APG73XN6TUEWRBB3WZOURQ, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.831349433 (24h27m11.302560172s) [noise=true , address=77.90.31.18:4807 , piece_id=WJ3AOAEE3IG3T3QQRFSKTOJCYVST4263Y5UN6XVOVTSO5ALOVTCQ, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.831805731 (24h27m11.30301647s) [noise=true , address=145.79.227.43:4158 , piece_id=74VKW6UWNPLSDKXJ6AG6TTUSDBCW4K4BAGMFIDNIV55AIWRBVB2Q, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.832172379 (24h27m11.303383118s) [noise=true , address=145.79.227.39:4014 , piece_id=ZLJ7L4OXXXNMJP2NTDU5HNE7NDCFL6JNWLXB6KVZHHFZXF5AT4CA, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.833230595 (24h27m11.304441334s) [noise=true , address=145.79.227.34:4905 , piece_id=ULLE3QB2QSCSYQWZ4OSBDJHC3X7AOHJGRWRA4DUXAMOYBDJRYWRQ, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.833972006 (24h27m11.305182755s) [noise=true , address=145.79.227.10:4401 , piece_id=TLP7B6MNA7W35UJQKW5SW6F43YXXMUFXOUKW4DKSOWOPIKLSSVUA, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:35.930150729 (24h27m12.401361488s) [noise=true , address=77.90.31.19:4843 , piece_id=36QLYXSVN6L2FDKVX4WS7MRTK54XLB2JUTW2NKHCE7E6KZGTLUKQ, seg_pos={0 0}, upload=21507]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: storage node overloaded, request limit: 2000 Dec 4 17:22:36.479057771 (24h27m12.9502685s) [upload=21507]: Next returned nil segment Dec 4 17:22:36.479122765 (24h27m12.950333494s) [upload=21507]: Waiting for error group managing segments... Dec 4 17:22:36.876009453 (24h27m13.347220192s) [noise=true , address=77.90.31.21:4878 , piece_id=GHU3SZUB5WAMW37CYCUFJ6HOPMXRNZGIJYDC4DJ3WOHV7YYWCVEA, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:36.891015988 (24h27m13.362226717s) [noise=true , address=77.90.31.14:4661 , piece_id=D5L5WWE5DGEYKP5M2NM74EGSX5BLEP4DX5OBM2Z4Q2QTCSGA4WXQ, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:36.943731919 (24h27m13.414942658s) [noise=true , address=77.90.31.10:4520 , piece_id=DGMWUPIGVLYJ4IH2CXCHA3U7FHDJF7APG73XN6TUEWRBB3WZOURQ, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:36.981669560 (24h27m13.452880289s) [noise=true , address=145.79.227.16:4910 , piece_id=36JF4CMQH5JO66BY5A53QNZQ6RIFTHZ6MD2QLBGHLHML5VBFGK7A, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:36.981785620 (24h27m13.452996359s) [noise=true , address=145.79.227.21:4475 , piece_id=YZRHRIA7WWLZKDX3BZ2FSUOKJ3MX7D7Q2GB6CHKVQKGXPQCUNC6A, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:36.984955717 (24h27m13.456166456s) [noise=true , address=77.90.31.18:4807 , piece_id=WJ3AOAEE3IG3T3QQRFSKTOJCYVST4263Y5UN6XVOVTSO5ALOVTCQ, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:36.994452571 (24h27m13.4656633s) [noise=true , address=145.79.227.34:4905 , piece_id=ULLE3QB2QSCSYQWZ4OSBDJHC3X7AOHJGRWRA4DUXAMOYBDJRYWRQ, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:37.011055396 (24h27m13.482266125s) [noise=true , address=145.79.227.38:4978 , piece_id=JLYEPL2WLUJIIUHI3FSOWIT5QG3ZQG3NNOZZ3TNMAOEWLZJNN6AA, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:37.021499513 (24h27m13.492710252s) [noise=true , address=145.79.227.8:4329 , piece_id=FBD34B6US36TX6YHN5UEBKOWIVOOP4G7B6SSYKN6Q3J5YNE7OBHA, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:37.043392031 (24h27m13.51460277s) [noise=true , address=145.79.227.39:4014 , piece_id=ZLJ7L4OXXXNMJP2NTDU5HNE7NDCFL6JNWLXB6KVZHHFZXF5AT4CA, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:37.047232213 (24h27m13.518442952s) [noise=true , address=145.79.227.29:4942 , piece_id=QOOXF6G2OGBNJLA64B2YNVVJKAZSP6HM5ZIEI4ZFJKWZUSQAR3XA, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:37.315831267 (24h27m13.787042006s) [noise=true , address=145.79.227.10:4401 , piece_id=TLP7B6MNA7W35UJQKW5SW6F43YXXMUFXOUKW4DKSOWOPIKLSSVUA, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:37.332403453 (24h27m13.803614192s) [noise=true , address=145.79.227.43:4157 , piece_id=VWS32XVNWH4R5PZDDV4FGP5ONZQZNLMU5YYB7QYS43GDJSDFX2OQ, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:37.332432619 (24h27m13.803643358s) [seg_pos={0 0}, upload=21507]: Segment reached optimal threshold of 13 pieces. Dec 4 17:22:37.332650384 (24h27m13.803861123s) [noise=true , address=145.79.227.9:4366 , piece_id=GDCC3W66FFB27QD5MRCG2MBJXZBWXRIWG7A4L3XMFTIRNN2NYPUQ, seg_pos={0 0}, upload=21507]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy): protocol: context canceled Dec 4 17:22:37.332660022 (24h27m13.803870751s) [noise=true , address=145.79.227.9:4366 , piece_id=GDCC3W66FFB27QD5MRCG2MBJXZBWXRIWG7A4L3XMFTIRNN2NYPUQ, seg_pos={0 0}, upload=21507]: Long tail cancellation due to optimal threshold Dec 4 17:22:37.332942179 (24h27m13.804152938s) [noise=true , address=145.79.227.43:4158 , piece_id=74VKW6UWNPLSDKXJ6AG6TTUSDBCW4K4BAGMFIDNIV55AIWRBVB2Q, seg_pos={0 0}, upload=21507]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vJZrSvPvwebw9V2tTbvkPEmLeLjryBUnq9TL6NwdPsHU9xcq2): protocol: expected piece hash; context canceled; EOF Dec 4 17:22:37.332954993 (24h27m13.804165732s) [noise=true , address=145.79.227.43:4158 , piece_id=74VKW6UWNPLSDKXJ6AG6TTUSDBCW4K4BAGMFIDNIV55AIWRBVB2Q, seg_pos={0 0}, upload=21507]: Long tail cancellation due to optimal threshold Dec 4 17:22:37.332976635 (24h27m13.804187384s) [seg_pos={0 0}, upload=21507]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 4 17:22:37.333114427 (24h27m13.804325166s) [upload=21507]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 4 17:22:37.333121821 (24h27m13.80433255s) [upload=21507]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 4 17:22:37.958438444 (24h27m14.429649173s) [upload=21507]: Done uploading segments... Dec 4 17:22:37.958456208 (24h27m14.429666937s) [upload=21507]: Upload finished. err: <nil> Duration: 5.902644482s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 5.677887441s trace: https://jaeger.dev.storj.io/trace/6d15aafc5ce7b199 FAIL (2025-12-04 17:31:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, Piece ID: FBD34B6US36TX6YHN5UEBKOWIVOOP4G7B6SSYKN6Q3J5YNE7OBHA) context deadline exceeded; piecestore: (Node ID: 12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9, Piece ID: ULLE3QB2QSCSYQWZ4OSBDJHC3X7AOHJGRWRA4DUXAMOYBDJRYWRQ) context deadline exceeded; piecestore: (Node ID: 1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, Piece ID: TLP7B6MNA7W35UJQKW5SW6F43YXXMUFXOUKW4DKSOWOPIKLSSVUA) context deadline exceeded; piecestore: (Node ID: 17o881pUMuG7K2dv83k7RkrJCrmM7kgUiWesgam4GEw21KVUn7, Piece ID: QOOXF6G2OGBNJLA64B2YNVVJKAZSP6HM5ZIEI4ZFJKWZUSQAR3XA) context deadline exceeded; piecestore: (Node ID: 1ZrzADk2mKG4R9tjxUKjQeJvb4WmfjLksGFyvstHRfYbBC7XNz, Piece ID: VWS32XVNWH4R5PZDDV4FGP5ONZQZNLMU5YYB7QYS43GDJSDFX2OQ) context deadline exceeded ======LOGS====== Dec 4 17:22:34.320746327 (24h27m10.791957066s) [upload=21507]: Starting upload Dec 4 17:22:34.320860715 (24h27m10.792071454s) [upload=21507]: Done starting upload Dec 4 17:22:34.321208968 (24h27m10.792419707s) [upload=21507]: Uploading segments... Dec 4 17:22:34.321325259 (24h27m10.792536018s) [upload=21507]: Got next segment. Inline: false Dec 4 17:22:34.321394982 (24h27m10.792605711s) [upload=21507]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 17:22:34.321401375 (24h27m10.792612114s) [upload=21507]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 17:22:34.827776318 (24h27m11.298987057s) [seg_pos={0 0}, upload=21507]: Begin upload segment... Dec 4 17:22:34.827838226 (24h27m11.299048966s) [seg_pos={0 0}, upload=21507]: Done begin upload segment. Dec 4 17:22:34.827877361 (24h27m11.29908811s) [noise=true , address=77.90.31.14:4661 , piece_id=D5L5WWE5DGEYKP5M2NM74EGSX5BLEP4DX5OBM2Z4Q2QTCSGA4WXQ, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.827887630 (24h27m11.299098359s) [noise=true , address=77.90.31.21:4878 , piece_id=GHU3SZUB5WAMW37CYCUFJ6HOPMXRNZGIJYDC4DJ3WOHV7YYWCVEA, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.828727861 (24h27m11.2999386s) [noise=true , address=145.79.227.29:4942 , piece_id=QOOXF6G2OGBNJLA64B2YNVVJKAZSP6HM5ZIEI4ZFJKWZUSQAR3XA, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.828821058 (24h27m11.300031807s) [noise=true , address=145.79.227.16:4910 , piece_id=36JF4CMQH5JO66BY5A53QNZQ6RIFTHZ6MD2QLBGHLHML5VBFGK7A, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.829248923 (24h27m11.300459652s) [noise=true , address=145.79.227.43:4157 , piece_id=VWS32XVNWH4R5PZDDV4FGP5ONZQZNLMU5YYB7QYS43GDJSDFX2OQ, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.829618296 (24h27m11.300829035s) [noise=true , address=145.79.227.38:4978 , piece_id=JLYEPL2WLUJIIUHI3FSOWIT5QG3ZQG3NNOZZ3TNMAOEWLZJNN6AA, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.830015262 (24h27m11.301226001s) [noise=true , address=77.90.31.19:4843 , piece_id=36QLYXSVN6L2FDKVX4WS7MRTK54XLB2JUTW2NKHCE7E6KZGTLUKQ, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.830278002 (24h27m11.301488741s) [noise=true , address=145.79.227.21:4475 , piece_id=YZRHRIA7WWLZKDX3BZ2FSUOKJ3MX7D7Q2GB6CHKVQKGXPQCUNC6A, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.830415063 (24h27m11.301625802s) [noise=true , address=145.79.227.8:4329 , piece_id=FBD34B6US36TX6YHN5UEBKOWIVOOP4G7B6SSYKN6Q3J5YNE7OBHA, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.830692902 (24h27m11.301903641s) [noise=true , address=145.79.227.9:4366 , piece_id=GDCC3W66FFB27QD5MRCG2MBJXZBWXRIWG7A4L3XMFTIRNN2NYPUQ, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.830811337 (24h27m11.302022076s) [noise=true , address=77.90.31.10:4520 , piece_id=DGMWUPIGVLYJ4IH2CXCHA3U7FHDJF7APG73XN6TUEWRBB3WZOURQ, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.831349433 (24h27m11.302560172s) [noise=true , address=77.90.31.18:4807 , piece_id=WJ3AOAEE3IG3T3QQRFSKTOJCYVST4263Y5UN6XVOVTSO5ALOVTCQ, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.831805731 (24h27m11.30301647s) [noise=true , address=145.79.227.43:4158 , piece_id=74VKW6UWNPLSDKXJ6AG6TTUSDBCW4K4BAGMFIDNIV55AIWRBVB2Q, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.832172379 (24h27m11.303383118s) [noise=true , address=145.79.227.39:4014 , piece_id=ZLJ7L4OXXXNMJP2NTDU5HNE7NDCFL6JNWLXB6KVZHHFZXF5AT4CA, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.833230595 (24h27m11.304441334s) [noise=true , address=145.79.227.34:4905 , piece_id=ULLE3QB2QSCSYQWZ4OSBDJHC3X7AOHJGRWRA4DUXAMOYBDJRYWRQ, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.833972006 (24h27m11.305182755s) [noise=true , address=145.79.227.10:4401 , piece_id=TLP7B6MNA7W35UJQKW5SW6F43YXXMUFXOUKW4DKSOWOPIKLSSVUA, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:35.930150729 (24h27m12.401361488s) [noise=true , address=77.90.31.19:4843 , piece_id=36QLYXSVN6L2FDKVX4WS7MRTK54XLB2JUTW2NKHCE7E6KZGTLUKQ, seg_pos={0 0}, upload=21507]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: storage node overloaded, request limit: 2000 Dec 4 17:22:36.479057771 (24h27m12.9502685s) [upload=21507]: Next returned nil segment Dec 4 17:22:36.479122765 (24h27m12.950333494s) [upload=21507]: Waiting for error group managing segments... Dec 4 17:22:36.876009453 (24h27m13.347220192s) [noise=true , address=77.90.31.21:4878 , piece_id=GHU3SZUB5WAMW37CYCUFJ6HOPMXRNZGIJYDC4DJ3WOHV7YYWCVEA, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:36.891015988 (24h27m13.362226717s) [noise=true , address=77.90.31.14:4661 , piece_id=D5L5WWE5DGEYKP5M2NM74EGSX5BLEP4DX5OBM2Z4Q2QTCSGA4WXQ, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:36.943731919 (24h27m13.414942658s) [noise=true , address=77.90.31.10:4520 , piece_id=DGMWUPIGVLYJ4IH2CXCHA3U7FHDJF7APG73XN6TUEWRBB3WZOURQ, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:36.981669560 (24h27m13.452880289s) [noise=true , address=145.79.227.16:4910 , piece_id=36JF4CMQH5JO66BY5A53QNZQ6RIFTHZ6MD2QLBGHLHML5VBFGK7A, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:36.981785620 (24h27m13.452996359s) [noise=true , address=145.79.227.21:4475 , piece_id=YZRHRIA7WWLZKDX3BZ2FSUOKJ3MX7D7Q2GB6CHKVQKGXPQCUNC6A, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:36.984955717 (24h27m13.456166456s) [noise=true , address=77.90.31.18:4807 , piece_id=WJ3AOAEE3IG3T3QQRFSKTOJCYVST4263Y5UN6XVOVTSO5ALOVTCQ, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:36.994452571 (24h27m13.4656633s) [noise=true , address=145.79.227.34:4905 , piece_id=ULLE3QB2QSCSYQWZ4OSBDJHC3X7AOHJGRWRA4DUXAMOYBDJRYWRQ, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:37.011055396 (24h27m13.482266125s) [noise=true , address=145.79.227.38:4978 , piece_id=JLYEPL2WLUJIIUHI3FSOWIT5QG3ZQG3NNOZZ3TNMAOEWLZJNN6AA, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:37.021499513 (24h27m13.492710252s) [noise=true , address=145.79.227.8:4329 , piece_id=FBD34B6US36TX6YHN5UEBKOWIVOOP4G7B6SSYKN6Q3J5YNE7OBHA, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:37.043392031 (24h27m13.51460277s) [noise=true , address=145.79.227.39:4014 , piece_id=ZLJ7L4OXXXNMJP2NTDU5HNE7NDCFL6JNWLXB6KVZHHFZXF5AT4CA, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:37.047232213 (24h27m13.518442952s) [noise=true , address=145.79.227.29:4942 , piece_id=QOOXF6G2OGBNJLA64B2YNVVJKAZSP6HM5ZIEI4ZFJKWZUSQAR3XA, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:37.315831267 (24h27m13.787042006s) [noise=true , address=145.79.227.10:4401 , piece_id=TLP7B6MNA7W35UJQKW5SW6F43YXXMUFXOUKW4DKSOWOPIKLSSVUA, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:37.332403453 (24h27m13.803614192s) [noise=true , address=145.79.227.43:4157 , piece_id=VWS32XVNWH4R5PZDDV4FGP5ONZQZNLMU5YYB7QYS43GDJSDFX2OQ, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:37.332432619 (24h27m13.803643358s) [seg_pos={0 0}, upload=21507]: Segment reached optimal threshold of 13 pieces. Dec 4 17:22:37.332650384 (24h27m13.803861123s) [noise=true , address=145.79.227.9:4366 , piece_id=GDCC3W66FFB27QD5MRCG2MBJXZBWXRIWG7A4L3XMFTIRNN2NYPUQ, seg_pos={0 0}, upload=21507]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy): protocol: context canceled Dec 4 17:22:37.332660022 (24h27m13.803870751s) [noise=true , address=145.79.227.9:4366 , piece_id=GDCC3W66FFB27QD5MRCG2MBJXZBWXRIWG7A4L3XMFTIRNN2NYPUQ, seg_pos={0 0}, upload=21507]: Long tail cancellation due to optimal threshold Dec 4 17:22:37.332942179 (24h27m13.804152938s) [noise=true , address=145.79.227.43:4158 , piece_id=74VKW6UWNPLSDKXJ6AG6TTUSDBCW4K4BAGMFIDNIV55AIWRBVB2Q, seg_pos={0 0}, upload=21507]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vJZrSvPvwebw9V2tTbvkPEmLeLjryBUnq9TL6NwdPsHU9xcq2): protocol: expected piece hash; context canceled; EOF Dec 4 17:22:37.332954993 (24h27m13.804165732s) [noise=true , address=145.79.227.43:4158 , piece_id=74VKW6UWNPLSDKXJ6AG6TTUSDBCW4K4BAGMFIDNIV55AIWRBVB2Q, seg_pos={0 0}, upload=21507]: Long tail cancellation due to optimal threshold Dec 4 17:22:37.332976635 (24h27m13.804187384s) [seg_pos={0 0}, upload=21507]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 4 17:22:37.333114427 (24h27m13.804325166s) [upload=21507]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 4 17:22:37.333121821 (24h27m13.80433255s) [upload=21507]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 4 17:22:37.958438444 (24h27m14.429649173s) [upload=21507]: Done uploading segments... Dec 4 17:22:37.958456208 (24h27m14.429666937s) [upload=21507]: Upload finished. err: <nil> Duration: 5.677913699s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [......................X.....................................] duration: 3.512017562s trace: https://jaeger.dev.storj.io/trace/1f3e769edbb82b7a FAIL (2025-12-04 17:31:13 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.512045775s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 5.830927997s trace: https://jaeger.dev.storj.io/trace/6d15aafc5ce7b199 FAIL (2025-12-04 17:30:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, Piece ID: FBD34B6US36TX6YHN5UEBKOWIVOOP4G7B6SSYKN6Q3J5YNE7OBHA) context deadline exceeded; piecestore: (Node ID: 12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9, Piece ID: ULLE3QB2QSCSYQWZ4OSBDJHC3X7AOHJGRWRA4DUXAMOYBDJRYWRQ) context deadline exceeded; piecestore: (Node ID: 1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, Piece ID: TLP7B6MNA7W35UJQKW5SW6F43YXXMUFXOUKW4DKSOWOPIKLSSVUA) context deadline exceeded; piecestore: (Node ID: 17o881pUMuG7K2dv83k7RkrJCrmM7kgUiWesgam4GEw21KVUn7, Piece ID: QOOXF6G2OGBNJLA64B2YNVVJKAZSP6HM5ZIEI4ZFJKWZUSQAR3XA) context deadline exceeded; piecestore: (Node ID: 1ZrzADk2mKG4R9tjxUKjQeJvb4WmfjLksGFyvstHRfYbBC7XNz, Piece ID: VWS32XVNWH4R5PZDDV4FGP5ONZQZNLMU5YYB7QYS43GDJSDFX2OQ) context deadline exceeded ======LOGS====== Dec 4 17:22:34.320746327 (24h27m10.791957066s) [upload=21507]: Starting upload Dec 4 17:22:34.320860715 (24h27m10.792071454s) [upload=21507]: Done starting upload Dec 4 17:22:34.321208968 (24h27m10.792419707s) [upload=21507]: Uploading segments... Dec 4 17:22:34.321325259 (24h27m10.792536018s) [upload=21507]: Got next segment. Inline: false Dec 4 17:22:34.321394982 (24h27m10.792605711s) [upload=21507]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 17:22:34.321401375 (24h27m10.792612114s) [upload=21507]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 17:22:34.827776318 (24h27m11.298987057s) [seg_pos={0 0}, upload=21507]: Begin upload segment... Dec 4 17:22:34.827838226 (24h27m11.299048966s) [seg_pos={0 0}, upload=21507]: Done begin upload segment. Dec 4 17:22:34.827877361 (24h27m11.29908811s) [noise=true , address=77.90.31.14:4661 , piece_id=D5L5WWE5DGEYKP5M2NM74EGSX5BLEP4DX5OBM2Z4Q2QTCSGA4WXQ, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.827887630 (24h27m11.299098359s) [noise=true , address=77.90.31.21:4878 , piece_id=GHU3SZUB5WAMW37CYCUFJ6HOPMXRNZGIJYDC4DJ3WOHV7YYWCVEA, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.828727861 (24h27m11.2999386s) [noise=true , address=145.79.227.29:4942 , piece_id=QOOXF6G2OGBNJLA64B2YNVVJKAZSP6HM5ZIEI4ZFJKWZUSQAR3XA, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.828821058 (24h27m11.300031807s) [noise=true , address=145.79.227.16:4910 , piece_id=36JF4CMQH5JO66BY5A53QNZQ6RIFTHZ6MD2QLBGHLHML5VBFGK7A, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.829248923 (24h27m11.300459652s) [noise=true , address=145.79.227.43:4157 , piece_id=VWS32XVNWH4R5PZDDV4FGP5ONZQZNLMU5YYB7QYS43GDJSDFX2OQ, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.829618296 (24h27m11.300829035s) [noise=true , address=145.79.227.38:4978 , piece_id=JLYEPL2WLUJIIUHI3FSOWIT5QG3ZQG3NNOZZ3TNMAOEWLZJNN6AA, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.830015262 (24h27m11.301226001s) [noise=true , address=77.90.31.19:4843 , piece_id=36QLYXSVN6L2FDKVX4WS7MRTK54XLB2JUTW2NKHCE7E6KZGTLUKQ, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.830278002 (24h27m11.301488741s) [noise=true , address=145.79.227.21:4475 , piece_id=YZRHRIA7WWLZKDX3BZ2FSUOKJ3MX7D7Q2GB6CHKVQKGXPQCUNC6A, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.830415063 (24h27m11.301625802s) [noise=true , address=145.79.227.8:4329 , piece_id=FBD34B6US36TX6YHN5UEBKOWIVOOP4G7B6SSYKN6Q3J5YNE7OBHA, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.830692902 (24h27m11.301903641s) [noise=true , address=145.79.227.9:4366 , piece_id=GDCC3W66FFB27QD5MRCG2MBJXZBWXRIWG7A4L3XMFTIRNN2NYPUQ, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.830811337 (24h27m11.302022076s) [noise=true , address=77.90.31.10:4520 , piece_id=DGMWUPIGVLYJ4IH2CXCHA3U7FHDJF7APG73XN6TUEWRBB3WZOURQ, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.831349433 (24h27m11.302560172s) [noise=true , address=77.90.31.18:4807 , piece_id=WJ3AOAEE3IG3T3QQRFSKTOJCYVST4263Y5UN6XVOVTSO5ALOVTCQ, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.831805731 (24h27m11.30301647s) [noise=true , address=145.79.227.43:4158 , piece_id=74VKW6UWNPLSDKXJ6AG6TTUSDBCW4K4BAGMFIDNIV55AIWRBVB2Q, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.832172379 (24h27m11.303383118s) [noise=true , address=145.79.227.39:4014 , piece_id=ZLJ7L4OXXXNMJP2NTDU5HNE7NDCFL6JNWLXB6KVZHHFZXF5AT4CA, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.833230595 (24h27m11.304441334s) [noise=true , address=145.79.227.34:4905 , piece_id=ULLE3QB2QSCSYQWZ4OSBDJHC3X7AOHJGRWRA4DUXAMOYBDJRYWRQ, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.833972006 (24h27m11.305182755s) [noise=true , address=145.79.227.10:4401 , piece_id=TLP7B6MNA7W35UJQKW5SW6F43YXXMUFXOUKW4DKSOWOPIKLSSVUA, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:35.930150729 (24h27m12.401361488s) [noise=true , address=77.90.31.19:4843 , piece_id=36QLYXSVN6L2FDKVX4WS7MRTK54XLB2JUTW2NKHCE7E6KZGTLUKQ, seg_pos={0 0}, upload=21507]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: storage node overloaded, request limit: 2000 Dec 4 17:22:36.479057771 (24h27m12.9502685s) [upload=21507]: Next returned nil segment Dec 4 17:22:36.479122765 (24h27m12.950333494s) [upload=21507]: Waiting for error group managing segments... Dec 4 17:22:36.876009453 (24h27m13.347220192s) [noise=true , address=77.90.31.21:4878 , piece_id=GHU3SZUB5WAMW37CYCUFJ6HOPMXRNZGIJYDC4DJ3WOHV7YYWCVEA, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:36.891015988 (24h27m13.362226717s) [noise=true , address=77.90.31.14:4661 , piece_id=D5L5WWE5DGEYKP5M2NM74EGSX5BLEP4DX5OBM2Z4Q2QTCSGA4WXQ, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:36.943731919 (24h27m13.414942658s) [noise=true , address=77.90.31.10:4520 , piece_id=DGMWUPIGVLYJ4IH2CXCHA3U7FHDJF7APG73XN6TUEWRBB3WZOURQ, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:36.981669560 (24h27m13.452880289s) [noise=true , address=145.79.227.16:4910 , piece_id=36JF4CMQH5JO66BY5A53QNZQ6RIFTHZ6MD2QLBGHLHML5VBFGK7A, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:36.981785620 (24h27m13.452996359s) [noise=true , address=145.79.227.21:4475 , piece_id=YZRHRIA7WWLZKDX3BZ2FSUOKJ3MX7D7Q2GB6CHKVQKGXPQCUNC6A, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:36.984955717 (24h27m13.456166456s) [noise=true , address=77.90.31.18:4807 , piece_id=WJ3AOAEE3IG3T3QQRFSKTOJCYVST4263Y5UN6XVOVTSO5ALOVTCQ, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:36.994452571 (24h27m13.4656633s) [noise=true , address=145.79.227.34:4905 , piece_id=ULLE3QB2QSCSYQWZ4OSBDJHC3X7AOHJGRWRA4DUXAMOYBDJRYWRQ, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:37.011055396 (24h27m13.482266125s) [noise=true , address=145.79.227.38:4978 , piece_id=JLYEPL2WLUJIIUHI3FSOWIT5QG3ZQG3NNOZZ3TNMAOEWLZJNN6AA, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:37.021499513 (24h27m13.492710252s) [noise=true , address=145.79.227.8:4329 , piece_id=FBD34B6US36TX6YHN5UEBKOWIVOOP4G7B6SSYKN6Q3J5YNE7OBHA, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:37.043392031 (24h27m13.51460277s) [noise=true , address=145.79.227.39:4014 , piece_id=ZLJ7L4OXXXNMJP2NTDU5HNE7NDCFL6JNWLXB6KVZHHFZXF5AT4CA, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:37.047232213 (24h27m13.518442952s) [noise=true , address=145.79.227.29:4942 , piece_id=QOOXF6G2OGBNJLA64B2YNVVJKAZSP6HM5ZIEI4ZFJKWZUSQAR3XA, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:37.315831267 (24h27m13.787042006s) [noise=true , address=145.79.227.10:4401 , piece_id=TLP7B6MNA7W35UJQKW5SW6F43YXXMUFXOUKW4DKSOWOPIKLSSVUA, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:37.332403453 (24h27m13.803614192s) [noise=true , address=145.79.227.43:4157 , piece_id=VWS32XVNWH4R5PZDDV4FGP5ONZQZNLMU5YYB7QYS43GDJSDFX2OQ, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:37.332432619 (24h27m13.803643358s) [seg_pos={0 0}, upload=21507]: Segment reached optimal threshold of 13 pieces. Dec 4 17:22:37.332650384 (24h27m13.803861123s) [noise=true , address=145.79.227.9:4366 , piece_id=GDCC3W66FFB27QD5MRCG2MBJXZBWXRIWG7A4L3XMFTIRNN2NYPUQ, seg_pos={0 0}, upload=21507]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy): protocol: context canceled Dec 4 17:22:37.332660022 (24h27m13.803870751s) [noise=true , address=145.79.227.9:4366 , piece_id=GDCC3W66FFB27QD5MRCG2MBJXZBWXRIWG7A4L3XMFTIRNN2NYPUQ, seg_pos={0 0}, upload=21507]: Long tail cancellation due to optimal threshold Dec 4 17:22:37.332942179 (24h27m13.804152938s) [noise=true , address=145.79.227.43:4158 , piece_id=74VKW6UWNPLSDKXJ6AG6TTUSDBCW4K4BAGMFIDNIV55AIWRBVB2Q, seg_pos={0 0}, upload=21507]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vJZrSvPvwebw9V2tTbvkPEmLeLjryBUnq9TL6NwdPsHU9xcq2): protocol: expected piece hash; context canceled; EOF Dec 4 17:22:37.332954993 (24h27m13.804165732s) [noise=true , address=145.79.227.43:4158 , piece_id=74VKW6UWNPLSDKXJ6AG6TTUSDBCW4K4BAGMFIDNIV55AIWRBVB2Q, seg_pos={0 0}, upload=21507]: Long tail cancellation due to optimal threshold Dec 4 17:22:37.332976635 (24h27m13.804187384s) [seg_pos={0 0}, upload=21507]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 4 17:22:37.333114427 (24h27m13.804325166s) [upload=21507]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 4 17:22:37.333121821 (24h27m13.80433255s) [upload=21507]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 4 17:22:37.958438444 (24h27m14.429649173s) [upload=21507]: Done uploading segments... Dec 4 17:22:37.958456208 (24h27m14.429666937s) [upload=21507]: Upload finished. err: <nil> Duration: 5.830974907s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.......................X....................................] duration: 3.496287117s trace: https://jaeger.dev.storj.io/trace/1f3e769edbb82b7a FAIL (2025-12-04 17:30:04 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.496321613s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 5.717410791s trace: https://jaeger.dev.storj.io/trace/6d15aafc5ce7b199 FAIL (2025-12-04 17:29:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, Piece ID: FBD34B6US36TX6YHN5UEBKOWIVOOP4G7B6SSYKN6Q3J5YNE7OBHA) context deadline exceeded; piecestore: (Node ID: 12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9, Piece ID: ULLE3QB2QSCSYQWZ4OSBDJHC3X7AOHJGRWRA4DUXAMOYBDJRYWRQ) context deadline exceeded; piecestore: (Node ID: 1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, Piece ID: TLP7B6MNA7W35UJQKW5SW6F43YXXMUFXOUKW4DKSOWOPIKLSSVUA) context deadline exceeded; piecestore: (Node ID: 17o881pUMuG7K2dv83k7RkrJCrmM7kgUiWesgam4GEw21KVUn7, Piece ID: QOOXF6G2OGBNJLA64B2YNVVJKAZSP6HM5ZIEI4ZFJKWZUSQAR3XA) context deadline exceeded; piecestore: (Node ID: 1ZrzADk2mKG4R9tjxUKjQeJvb4WmfjLksGFyvstHRfYbBC7XNz, Piece ID: VWS32XVNWH4R5PZDDV4FGP5ONZQZNLMU5YYB7QYS43GDJSDFX2OQ) context deadline exceeded ======LOGS====== Dec 4 17:22:34.320746327 (24h27m10.791957066s) [upload=21507]: Starting upload Dec 4 17:22:34.320860715 (24h27m10.792071454s) [upload=21507]: Done starting upload Dec 4 17:22:34.321208968 (24h27m10.792419707s) [upload=21507]: Uploading segments... Dec 4 17:22:34.321325259 (24h27m10.792536018s) [upload=21507]: Got next segment. Inline: false Dec 4 17:22:34.321394982 (24h27m10.792605711s) [upload=21507]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 17:22:34.321401375 (24h27m10.792612114s) [upload=21507]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 17:22:34.827776318 (24h27m11.298987057s) [seg_pos={0 0}, upload=21507]: Begin upload segment... Dec 4 17:22:34.827838226 (24h27m11.299048966s) [seg_pos={0 0}, upload=21507]: Done begin upload segment. Dec 4 17:22:34.827877361 (24h27m11.29908811s) [noise=true , address=77.90.31.14:4661 , piece_id=D5L5WWE5DGEYKP5M2NM74EGSX5BLEP4DX5OBM2Z4Q2QTCSGA4WXQ, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.827887630 (24h27m11.299098359s) [noise=true , address=77.90.31.21:4878 , piece_id=GHU3SZUB5WAMW37CYCUFJ6HOPMXRNZGIJYDC4DJ3WOHV7YYWCVEA, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.828727861 (24h27m11.2999386s) [noise=true , address=145.79.227.29:4942 , piece_id=QOOXF6G2OGBNJLA64B2YNVVJKAZSP6HM5ZIEI4ZFJKWZUSQAR3XA, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.828821058 (24h27m11.300031807s) [noise=true , address=145.79.227.16:4910 , piece_id=36JF4CMQH5JO66BY5A53QNZQ6RIFTHZ6MD2QLBGHLHML5VBFGK7A, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.829248923 (24h27m11.300459652s) [noise=true , address=145.79.227.43:4157 , piece_id=VWS32XVNWH4R5PZDDV4FGP5ONZQZNLMU5YYB7QYS43GDJSDFX2OQ, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.829618296 (24h27m11.300829035s) [noise=true , address=145.79.227.38:4978 , piece_id=JLYEPL2WLUJIIUHI3FSOWIT5QG3ZQG3NNOZZ3TNMAOEWLZJNN6AA, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.830015262 (24h27m11.301226001s) [noise=true , address=77.90.31.19:4843 , piece_id=36QLYXSVN6L2FDKVX4WS7MRTK54XLB2JUTW2NKHCE7E6KZGTLUKQ, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.830278002 (24h27m11.301488741s) [noise=true , address=145.79.227.21:4475 , piece_id=YZRHRIA7WWLZKDX3BZ2FSUOKJ3MX7D7Q2GB6CHKVQKGXPQCUNC6A, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.830415063 (24h27m11.301625802s) [noise=true , address=145.79.227.8:4329 , piece_id=FBD34B6US36TX6YHN5UEBKOWIVOOP4G7B6SSYKN6Q3J5YNE7OBHA, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.830692902 (24h27m11.301903641s) [noise=true , address=145.79.227.9:4366 , piece_id=GDCC3W66FFB27QD5MRCG2MBJXZBWXRIWG7A4L3XMFTIRNN2NYPUQ, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.830811337 (24h27m11.302022076s) [noise=true , address=77.90.31.10:4520 , piece_id=DGMWUPIGVLYJ4IH2CXCHA3U7FHDJF7APG73XN6TUEWRBB3WZOURQ, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.831349433 (24h27m11.302560172s) [noise=true , address=77.90.31.18:4807 , piece_id=WJ3AOAEE3IG3T3QQRFSKTOJCYVST4263Y5UN6XVOVTSO5ALOVTCQ, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.831805731 (24h27m11.30301647s) [noise=true , address=145.79.227.43:4158 , piece_id=74VKW6UWNPLSDKXJ6AG6TTUSDBCW4K4BAGMFIDNIV55AIWRBVB2Q, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.832172379 (24h27m11.303383118s) [noise=true , address=145.79.227.39:4014 , piece_id=ZLJ7L4OXXXNMJP2NTDU5HNE7NDCFL6JNWLXB6KVZHHFZXF5AT4CA, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.833230595 (24h27m11.304441334s) [noise=true , address=145.79.227.34:4905 , piece_id=ULLE3QB2QSCSYQWZ4OSBDJHC3X7AOHJGRWRA4DUXAMOYBDJRYWRQ, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.833972006 (24h27m11.305182755s) [noise=true , address=145.79.227.10:4401 , piece_id=TLP7B6MNA7W35UJQKW5SW6F43YXXMUFXOUKW4DKSOWOPIKLSSVUA, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:35.930150729 (24h27m12.401361488s) [noise=true , address=77.90.31.19:4843 , piece_id=36QLYXSVN6L2FDKVX4WS7MRTK54XLB2JUTW2NKHCE7E6KZGTLUKQ, seg_pos={0 0}, upload=21507]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: storage node overloaded, request limit: 2000 Dec 4 17:22:36.479057771 (24h27m12.9502685s) [upload=21507]: Next returned nil segment Dec 4 17:22:36.479122765 (24h27m12.950333494s) [upload=21507]: Waiting for error group managing segments... Dec 4 17:22:36.876009453 (24h27m13.347220192s) [noise=true , address=77.90.31.21:4878 , piece_id=GHU3SZUB5WAMW37CYCUFJ6HOPMXRNZGIJYDC4DJ3WOHV7YYWCVEA, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:36.891015988 (24h27m13.362226717s) [noise=true , address=77.90.31.14:4661 , piece_id=D5L5WWE5DGEYKP5M2NM74EGSX5BLEP4DX5OBM2Z4Q2QTCSGA4WXQ, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:36.943731919 (24h27m13.414942658s) [noise=true , address=77.90.31.10:4520 , piece_id=DGMWUPIGVLYJ4IH2CXCHA3U7FHDJF7APG73XN6TUEWRBB3WZOURQ, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:36.981669560 (24h27m13.452880289s) [noise=true , address=145.79.227.16:4910 , piece_id=36JF4CMQH5JO66BY5A53QNZQ6RIFTHZ6MD2QLBGHLHML5VBFGK7A, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:36.981785620 (24h27m13.452996359s) [noise=true , address=145.79.227.21:4475 , piece_id=YZRHRIA7WWLZKDX3BZ2FSUOKJ3MX7D7Q2GB6CHKVQKGXPQCUNC6A, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:36.984955717 (24h27m13.456166456s) [noise=true , address=77.90.31.18:4807 , piece_id=WJ3AOAEE3IG3T3QQRFSKTOJCYVST4263Y5UN6XVOVTSO5ALOVTCQ, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:36.994452571 (24h27m13.4656633s) [noise=true , address=145.79.227.34:4905 , piece_id=ULLE3QB2QSCSYQWZ4OSBDJHC3X7AOHJGRWRA4DUXAMOYBDJRYWRQ, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:37.011055396 (24h27m13.482266125s) [noise=true , address=145.79.227.38:4978 , piece_id=JLYEPL2WLUJIIUHI3FSOWIT5QG3ZQG3NNOZZ3TNMAOEWLZJNN6AA, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:37.021499513 (24h27m13.492710252s) [noise=true , address=145.79.227.8:4329 , piece_id=FBD34B6US36TX6YHN5UEBKOWIVOOP4G7B6SSYKN6Q3J5YNE7OBHA, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:37.043392031 (24h27m13.51460277s) [noise=true , address=145.79.227.39:4014 , piece_id=ZLJ7L4OXXXNMJP2NTDU5HNE7NDCFL6JNWLXB6KVZHHFZXF5AT4CA, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:37.047232213 (24h27m13.518442952s) [noise=true , address=145.79.227.29:4942 , piece_id=QOOXF6G2OGBNJLA64B2YNVVJKAZSP6HM5ZIEI4ZFJKWZUSQAR3XA, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:37.315831267 (24h27m13.787042006s) [noise=true , address=145.79.227.10:4401 , piece_id=TLP7B6MNA7W35UJQKW5SW6F43YXXMUFXOUKW4DKSOWOPIKLSSVUA, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:37.332403453 (24h27m13.803614192s) [noise=true , address=145.79.227.43:4157 , piece_id=VWS32XVNWH4R5PZDDV4FGP5ONZQZNLMU5YYB7QYS43GDJSDFX2OQ, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:37.332432619 (24h27m13.803643358s) [seg_pos={0 0}, upload=21507]: Segment reached optimal threshold of 13 pieces. Dec 4 17:22:37.332650384 (24h27m13.803861123s) [noise=true , address=145.79.227.9:4366 , piece_id=GDCC3W66FFB27QD5MRCG2MBJXZBWXRIWG7A4L3XMFTIRNN2NYPUQ, seg_pos={0 0}, upload=21507]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy): protocol: context canceled Dec 4 17:22:37.332660022 (24h27m13.803870751s) [noise=true , address=145.79.227.9:4366 , piece_id=GDCC3W66FFB27QD5MRCG2MBJXZBWXRIWG7A4L3XMFTIRNN2NYPUQ, seg_pos={0 0}, upload=21507]: Long tail cancellation due to optimal threshold Dec 4 17:22:37.332942179 (24h27m13.804152938s) [noise=true , address=145.79.227.43:4158 , piece_id=74VKW6UWNPLSDKXJ6AG6TTUSDBCW4K4BAGMFIDNIV55AIWRBVB2Q, seg_pos={0 0}, upload=21507]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vJZrSvPvwebw9V2tTbvkPEmLeLjryBUnq9TL6NwdPsHU9xcq2): protocol: expected piece hash; context canceled; EOF Dec 4 17:22:37.332954993 (24h27m13.804165732s) [noise=true , address=145.79.227.43:4158 , piece_id=74VKW6UWNPLSDKXJ6AG6TTUSDBCW4K4BAGMFIDNIV55AIWRBVB2Q, seg_pos={0 0}, upload=21507]: Long tail cancellation due to optimal threshold Dec 4 17:22:37.332976635 (24h27m13.804187384s) [seg_pos={0 0}, upload=21507]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 4 17:22:37.333114427 (24h27m13.804325166s) [upload=21507]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 4 17:22:37.333121821 (24h27m13.80433255s) [upload=21507]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 4 17:22:37.958438444 (24h27m14.429649173s) [upload=21507]: Done uploading segments... Dec 4 17:22:37.958456208 (24h27m14.429666937s) [upload=21507]: Upload finished. err: <nil> Duration: 5.717440518s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [........................X...................................] duration: 3.399293456s trace: https://jaeger.dev.storj.io/trace/1f3e769edbb82b7a FAIL (2025-12-04 17:29:20 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.39931738s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 5.745114222s trace: https://jaeger.dev.storj.io/trace/6d15aafc5ce7b199 FAIL (2025-12-04 17:28:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, Piece ID: FBD34B6US36TX6YHN5UEBKOWIVOOP4G7B6SSYKN6Q3J5YNE7OBHA) context deadline exceeded; piecestore: (Node ID: 12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9, Piece ID: ULLE3QB2QSCSYQWZ4OSBDJHC3X7AOHJGRWRA4DUXAMOYBDJRYWRQ) context deadline exceeded; piecestore: (Node ID: 1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, Piece ID: TLP7B6MNA7W35UJQKW5SW6F43YXXMUFXOUKW4DKSOWOPIKLSSVUA) context deadline exceeded; piecestore: (Node ID: 17o881pUMuG7K2dv83k7RkrJCrmM7kgUiWesgam4GEw21KVUn7, Piece ID: QOOXF6G2OGBNJLA64B2YNVVJKAZSP6HM5ZIEI4ZFJKWZUSQAR3XA) context deadline exceeded; piecestore: (Node ID: 1ZrzADk2mKG4R9tjxUKjQeJvb4WmfjLksGFyvstHRfYbBC7XNz, Piece ID: VWS32XVNWH4R5PZDDV4FGP5ONZQZNLMU5YYB7QYS43GDJSDFX2OQ) context deadline exceeded ======LOGS====== Dec 4 17:22:34.320746327 (24h27m10.791957066s) [upload=21507]: Starting upload Dec 4 17:22:34.320860715 (24h27m10.792071454s) [upload=21507]: Done starting upload Dec 4 17:22:34.321208968 (24h27m10.792419707s) [upload=21507]: Uploading segments... Dec 4 17:22:34.321325259 (24h27m10.792536018s) [upload=21507]: Got next segment. Inline: false Dec 4 17:22:34.321394982 (24h27m10.792605711s) [upload=21507]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 17:22:34.321401375 (24h27m10.792612114s) [upload=21507]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 17:22:34.827776318 (24h27m11.298987057s) [seg_pos={0 0}, upload=21507]: Begin upload segment... Dec 4 17:22:34.827838226 (24h27m11.299048966s) [seg_pos={0 0}, upload=21507]: Done begin upload segment. Dec 4 17:22:34.827877361 (24h27m11.29908811s) [noise=true , address=77.90.31.14:4661 , piece_id=D5L5WWE5DGEYKP5M2NM74EGSX5BLEP4DX5OBM2Z4Q2QTCSGA4WXQ, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.827887630 (24h27m11.299098359s) [noise=true , address=77.90.31.21:4878 , piece_id=GHU3SZUB5WAMW37CYCUFJ6HOPMXRNZGIJYDC4DJ3WOHV7YYWCVEA, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.828727861 (24h27m11.2999386s) [noise=true , address=145.79.227.29:4942 , piece_id=QOOXF6G2OGBNJLA64B2YNVVJKAZSP6HM5ZIEI4ZFJKWZUSQAR3XA, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.828821058 (24h27m11.300031807s) [noise=true , address=145.79.227.16:4910 , piece_id=36JF4CMQH5JO66BY5A53QNZQ6RIFTHZ6MD2QLBGHLHML5VBFGK7A, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.829248923 (24h27m11.300459652s) [noise=true , address=145.79.227.43:4157 , piece_id=VWS32XVNWH4R5PZDDV4FGP5ONZQZNLMU5YYB7QYS43GDJSDFX2OQ, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.829618296 (24h27m11.300829035s) [noise=true , address=145.79.227.38:4978 , piece_id=JLYEPL2WLUJIIUHI3FSOWIT5QG3ZQG3NNOZZ3TNMAOEWLZJNN6AA, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.830015262 (24h27m11.301226001s) [noise=true , address=77.90.31.19:4843 , piece_id=36QLYXSVN6L2FDKVX4WS7MRTK54XLB2JUTW2NKHCE7E6KZGTLUKQ, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.830278002 (24h27m11.301488741s) [noise=true , address=145.79.227.21:4475 , piece_id=YZRHRIA7WWLZKDX3BZ2FSUOKJ3MX7D7Q2GB6CHKVQKGXPQCUNC6A, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.830415063 (24h27m11.301625802s) [noise=true , address=145.79.227.8:4329 , piece_id=FBD34B6US36TX6YHN5UEBKOWIVOOP4G7B6SSYKN6Q3J5YNE7OBHA, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.830692902 (24h27m11.301903641s) [noise=true , address=145.79.227.9:4366 , piece_id=GDCC3W66FFB27QD5MRCG2MBJXZBWXRIWG7A4L3XMFTIRNN2NYPUQ, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.830811337 (24h27m11.302022076s) [noise=true , address=77.90.31.10:4520 , piece_id=DGMWUPIGVLYJ4IH2CXCHA3U7FHDJF7APG73XN6TUEWRBB3WZOURQ, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.831349433 (24h27m11.302560172s) [noise=true , address=77.90.31.18:4807 , piece_id=WJ3AOAEE3IG3T3QQRFSKTOJCYVST4263Y5UN6XVOVTSO5ALOVTCQ, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.831805731 (24h27m11.30301647s) [noise=true , address=145.79.227.43:4158 , piece_id=74VKW6UWNPLSDKXJ6AG6TTUSDBCW4K4BAGMFIDNIV55AIWRBVB2Q, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.832172379 (24h27m11.303383118s) [noise=true , address=145.79.227.39:4014 , piece_id=ZLJ7L4OXXXNMJP2NTDU5HNE7NDCFL6JNWLXB6KVZHHFZXF5AT4CA, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.833230595 (24h27m11.304441334s) [noise=true , address=145.79.227.34:4905 , piece_id=ULLE3QB2QSCSYQWZ4OSBDJHC3X7AOHJGRWRA4DUXAMOYBDJRYWRQ, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.833972006 (24h27m11.305182755s) [noise=true , address=145.79.227.10:4401 , piece_id=TLP7B6MNA7W35UJQKW5SW6F43YXXMUFXOUKW4DKSOWOPIKLSSVUA, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:35.930150729 (24h27m12.401361488s) [noise=true , address=77.90.31.19:4843 , piece_id=36QLYXSVN6L2FDKVX4WS7MRTK54XLB2JUTW2NKHCE7E6KZGTLUKQ, seg_pos={0 0}, upload=21507]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: storage node overloaded, request limit: 2000 Dec 4 17:22:36.479057771 (24h27m12.9502685s) [upload=21507]: Next returned nil segment Dec 4 17:22:36.479122765 (24h27m12.950333494s) [upload=21507]: Waiting for error group managing segments... Dec 4 17:22:36.876009453 (24h27m13.347220192s) [noise=true , address=77.90.31.21:4878 , piece_id=GHU3SZUB5WAMW37CYCUFJ6HOPMXRNZGIJYDC4DJ3WOHV7YYWCVEA, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:36.891015988 (24h27m13.362226717s) [noise=true , address=77.90.31.14:4661 , piece_id=D5L5WWE5DGEYKP5M2NM74EGSX5BLEP4DX5OBM2Z4Q2QTCSGA4WXQ, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:36.943731919 (24h27m13.414942658s) [noise=true , address=77.90.31.10:4520 , piece_id=DGMWUPIGVLYJ4IH2CXCHA3U7FHDJF7APG73XN6TUEWRBB3WZOURQ, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:36.981669560 (24h27m13.452880289s) [noise=true , address=145.79.227.16:4910 , piece_id=36JF4CMQH5JO66BY5A53QNZQ6RIFTHZ6MD2QLBGHLHML5VBFGK7A, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:36.981785620 (24h27m13.452996359s) [noise=true , address=145.79.227.21:4475 , piece_id=YZRHRIA7WWLZKDX3BZ2FSUOKJ3MX7D7Q2GB6CHKVQKGXPQCUNC6A, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:36.984955717 (24h27m13.456166456s) [noise=true , address=77.90.31.18:4807 , piece_id=WJ3AOAEE3IG3T3QQRFSKTOJCYVST4263Y5UN6XVOVTSO5ALOVTCQ, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:36.994452571 (24h27m13.4656633s) [noise=true , address=145.79.227.34:4905 , piece_id=ULLE3QB2QSCSYQWZ4OSBDJHC3X7AOHJGRWRA4DUXAMOYBDJRYWRQ, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:37.011055396 (24h27m13.482266125s) [noise=true , address=145.79.227.38:4978 , piece_id=JLYEPL2WLUJIIUHI3FSOWIT5QG3ZQG3NNOZZ3TNMAOEWLZJNN6AA, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:37.021499513 (24h27m13.492710252s) [noise=true , address=145.79.227.8:4329 , piece_id=FBD34B6US36TX6YHN5UEBKOWIVOOP4G7B6SSYKN6Q3J5YNE7OBHA, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:37.043392031 (24h27m13.51460277s) [noise=true , address=145.79.227.39:4014 , piece_id=ZLJ7L4OXXXNMJP2NTDU5HNE7NDCFL6JNWLXB6KVZHHFZXF5AT4CA, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:37.047232213 (24h27m13.518442952s) [noise=true , address=145.79.227.29:4942 , piece_id=QOOXF6G2OGBNJLA64B2YNVVJKAZSP6HM5ZIEI4ZFJKWZUSQAR3XA, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:37.315831267 (24h27m13.787042006s) [noise=true , address=145.79.227.10:4401 , piece_id=TLP7B6MNA7W35UJQKW5SW6F43YXXMUFXOUKW4DKSOWOPIKLSSVUA, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:37.332403453 (24h27m13.803614192s) [noise=true , address=145.79.227.43:4157 , piece_id=VWS32XVNWH4R5PZDDV4FGP5ONZQZNLMU5YYB7QYS43GDJSDFX2OQ, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:37.332432619 (24h27m13.803643358s) [seg_pos={0 0}, upload=21507]: Segment reached optimal threshold of 13 pieces. Dec 4 17:22:37.332650384 (24h27m13.803861123s) [noise=true , address=145.79.227.9:4366 , piece_id=GDCC3W66FFB27QD5MRCG2MBJXZBWXRIWG7A4L3XMFTIRNN2NYPUQ, seg_pos={0 0}, upload=21507]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy): protocol: context canceled Dec 4 17:22:37.332660022 (24h27m13.803870751s) [noise=true , address=145.79.227.9:4366 , piece_id=GDCC3W66FFB27QD5MRCG2MBJXZBWXRIWG7A4L3XMFTIRNN2NYPUQ, seg_pos={0 0}, upload=21507]: Long tail cancellation due to optimal threshold Dec 4 17:22:37.332942179 (24h27m13.804152938s) [noise=true , address=145.79.227.43:4158 , piece_id=74VKW6UWNPLSDKXJ6AG6TTUSDBCW4K4BAGMFIDNIV55AIWRBVB2Q, seg_pos={0 0}, upload=21507]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vJZrSvPvwebw9V2tTbvkPEmLeLjryBUnq9TL6NwdPsHU9xcq2): protocol: expected piece hash; context canceled; EOF Dec 4 17:22:37.332954993 (24h27m13.804165732s) [noise=true , address=145.79.227.43:4158 , piece_id=74VKW6UWNPLSDKXJ6AG6TTUSDBCW4K4BAGMFIDNIV55AIWRBVB2Q, seg_pos={0 0}, upload=21507]: Long tail cancellation due to optimal threshold Dec 4 17:22:37.332976635 (24h27m13.804187384s) [seg_pos={0 0}, upload=21507]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 4 17:22:37.333114427 (24h27m13.804325166s) [upload=21507]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 4 17:22:37.333121821 (24h27m13.80433255s) [upload=21507]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 4 17:22:37.958438444 (24h27m14.429649173s) [upload=21507]: Done uploading segments... Dec 4 17:22:37.958456208 (24h27m14.429666937s) [upload=21507]: Upload finished. err: <nil> Duration: 5.745146404s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.........................X..................................] duration: 3.360300886s trace: https://jaeger.dev.storj.io/trace/1f3e769edbb82b7a FAIL (2025-12-04 17:28:44 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.360330562s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..........................X.................................] duration: 3.384948492s trace: https://jaeger.dev.storj.io/trace/1f3e769edbb82b7a FAIL (2025-12-04 17:28:26 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.384977878s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...........................X................................] duration: 3.533067558s trace: https://jaeger.dev.storj.io/trace/1f3e769edbb82b7a FAIL (2025-12-04 17:27:57 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.533097745s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 5.818296377s trace: https://jaeger.dev.storj.io/trace/6d15aafc5ce7b199 FAIL (2025-12-04 17:27:42 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, Piece ID: FBD34B6US36TX6YHN5UEBKOWIVOOP4G7B6SSYKN6Q3J5YNE7OBHA) context deadline exceeded; piecestore: (Node ID: 12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9, Piece ID: ULLE3QB2QSCSYQWZ4OSBDJHC3X7AOHJGRWRA4DUXAMOYBDJRYWRQ) context deadline exceeded; piecestore: (Node ID: 1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, Piece ID: TLP7B6MNA7W35UJQKW5SW6F43YXXMUFXOUKW4DKSOWOPIKLSSVUA) context deadline exceeded; piecestore: (Node ID: 17o881pUMuG7K2dv83k7RkrJCrmM7kgUiWesgam4GEw21KVUn7, Piece ID: QOOXF6G2OGBNJLA64B2YNVVJKAZSP6HM5ZIEI4ZFJKWZUSQAR3XA) context deadline exceeded; piecestore: (Node ID: 1ZrzADk2mKG4R9tjxUKjQeJvb4WmfjLksGFyvstHRfYbBC7XNz, Piece ID: VWS32XVNWH4R5PZDDV4FGP5ONZQZNLMU5YYB7QYS43GDJSDFX2OQ) context deadline exceeded ======LOGS====== Dec 4 17:22:34.320746327 (24h27m10.791957066s) [upload=21507]: Starting upload Dec 4 17:22:34.320860715 (24h27m10.792071454s) [upload=21507]: Done starting upload Dec 4 17:22:34.321208968 (24h27m10.792419707s) [upload=21507]: Uploading segments... Dec 4 17:22:34.321325259 (24h27m10.792536018s) [upload=21507]: Got next segment. Inline: false Dec 4 17:22:34.321394982 (24h27m10.792605711s) [upload=21507]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 17:22:34.321401375 (24h27m10.792612114s) [upload=21507]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 17:22:34.827776318 (24h27m11.298987057s) [seg_pos={0 0}, upload=21507]: Begin upload segment... Dec 4 17:22:34.827838226 (24h27m11.299048966s) [seg_pos={0 0}, upload=21507]: Done begin upload segment. Dec 4 17:22:34.827877361 (24h27m11.29908811s) [noise=true , address=77.90.31.14:4661 , piece_id=D5L5WWE5DGEYKP5M2NM74EGSX5BLEP4DX5OBM2Z4Q2QTCSGA4WXQ, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.827887630 (24h27m11.299098359s) [noise=true , address=77.90.31.21:4878 , piece_id=GHU3SZUB5WAMW37CYCUFJ6HOPMXRNZGIJYDC4DJ3WOHV7YYWCVEA, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.828727861 (24h27m11.2999386s) [noise=true , address=145.79.227.29:4942 , piece_id=QOOXF6G2OGBNJLA64B2YNVVJKAZSP6HM5ZIEI4ZFJKWZUSQAR3XA, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.828821058 (24h27m11.300031807s) [noise=true , address=145.79.227.16:4910 , piece_id=36JF4CMQH5JO66BY5A53QNZQ6RIFTHZ6MD2QLBGHLHML5VBFGK7A, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.829248923 (24h27m11.300459652s) [noise=true , address=145.79.227.43:4157 , piece_id=VWS32XVNWH4R5PZDDV4FGP5ONZQZNLMU5YYB7QYS43GDJSDFX2OQ, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.829618296 (24h27m11.300829035s) [noise=true , address=145.79.227.38:4978 , piece_id=JLYEPL2WLUJIIUHI3FSOWIT5QG3ZQG3NNOZZ3TNMAOEWLZJNN6AA, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.830015262 (24h27m11.301226001s) [noise=true , address=77.90.31.19:4843 , piece_id=36QLYXSVN6L2FDKVX4WS7MRTK54XLB2JUTW2NKHCE7E6KZGTLUKQ, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.830278002 (24h27m11.301488741s) [noise=true , address=145.79.227.21:4475 , piece_id=YZRHRIA7WWLZKDX3BZ2FSUOKJ3MX7D7Q2GB6CHKVQKGXPQCUNC6A, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.830415063 (24h27m11.301625802s) [noise=true , address=145.79.227.8:4329 , piece_id=FBD34B6US36TX6YHN5UEBKOWIVOOP4G7B6SSYKN6Q3J5YNE7OBHA, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.830692902 (24h27m11.301903641s) [noise=true , address=145.79.227.9:4366 , piece_id=GDCC3W66FFB27QD5MRCG2MBJXZBWXRIWG7A4L3XMFTIRNN2NYPUQ, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.830811337 (24h27m11.302022076s) [noise=true , address=77.90.31.10:4520 , piece_id=DGMWUPIGVLYJ4IH2CXCHA3U7FHDJF7APG73XN6TUEWRBB3WZOURQ, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.831349433 (24h27m11.302560172s) [noise=true , address=77.90.31.18:4807 , piece_id=WJ3AOAEE3IG3T3QQRFSKTOJCYVST4263Y5UN6XVOVTSO5ALOVTCQ, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.831805731 (24h27m11.30301647s) [noise=true , address=145.79.227.43:4158 , piece_id=74VKW6UWNPLSDKXJ6AG6TTUSDBCW4K4BAGMFIDNIV55AIWRBVB2Q, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.832172379 (24h27m11.303383118s) [noise=true , address=145.79.227.39:4014 , piece_id=ZLJ7L4OXXXNMJP2NTDU5HNE7NDCFL6JNWLXB6KVZHHFZXF5AT4CA, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.833230595 (24h27m11.304441334s) [noise=true , address=145.79.227.34:4905 , piece_id=ULLE3QB2QSCSYQWZ4OSBDJHC3X7AOHJGRWRA4DUXAMOYBDJRYWRQ, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.833972006 (24h27m11.305182755s) [noise=true , address=145.79.227.10:4401 , piece_id=TLP7B6MNA7W35UJQKW5SW6F43YXXMUFXOUKW4DKSOWOPIKLSSVUA, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:35.930150729 (24h27m12.401361488s) [noise=true , address=77.90.31.19:4843 , piece_id=36QLYXSVN6L2FDKVX4WS7MRTK54XLB2JUTW2NKHCE7E6KZGTLUKQ, seg_pos={0 0}, upload=21507]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: storage node overloaded, request limit: 2000 Dec 4 17:22:36.479057771 (24h27m12.9502685s) [upload=21507]: Next returned nil segment Dec 4 17:22:36.479122765 (24h27m12.950333494s) [upload=21507]: Waiting for error group managing segments... Dec 4 17:22:36.876009453 (24h27m13.347220192s) [noise=true , address=77.90.31.21:4878 , piece_id=GHU3SZUB5WAMW37CYCUFJ6HOPMXRNZGIJYDC4DJ3WOHV7YYWCVEA, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:36.891015988 (24h27m13.362226717s) [noise=true , address=77.90.31.14:4661 , piece_id=D5L5WWE5DGEYKP5M2NM74EGSX5BLEP4DX5OBM2Z4Q2QTCSGA4WXQ, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:36.943731919 (24h27m13.414942658s) [noise=true , address=77.90.31.10:4520 , piece_id=DGMWUPIGVLYJ4IH2CXCHA3U7FHDJF7APG73XN6TUEWRBB3WZOURQ, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:36.981669560 (24h27m13.452880289s) [noise=true , address=145.79.227.16:4910 , piece_id=36JF4CMQH5JO66BY5A53QNZQ6RIFTHZ6MD2QLBGHLHML5VBFGK7A, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:36.981785620 (24h27m13.452996359s) [noise=true , address=145.79.227.21:4475 , piece_id=YZRHRIA7WWLZKDX3BZ2FSUOKJ3MX7D7Q2GB6CHKVQKGXPQCUNC6A, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:36.984955717 (24h27m13.456166456s) [noise=true , address=77.90.31.18:4807 , piece_id=WJ3AOAEE3IG3T3QQRFSKTOJCYVST4263Y5UN6XVOVTSO5ALOVTCQ, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:36.994452571 (24h27m13.4656633s) [noise=true , address=145.79.227.34:4905 , piece_id=ULLE3QB2QSCSYQWZ4OSBDJHC3X7AOHJGRWRA4DUXAMOYBDJRYWRQ, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:37.011055396 (24h27m13.482266125s) [noise=true , address=145.79.227.38:4978 , piece_id=JLYEPL2WLUJIIUHI3FSOWIT5QG3ZQG3NNOZZ3TNMAOEWLZJNN6AA, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:37.021499513 (24h27m13.492710252s) [noise=true , address=145.79.227.8:4329 , piece_id=FBD34B6US36TX6YHN5UEBKOWIVOOP4G7B6SSYKN6Q3J5YNE7OBHA, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:37.043392031 (24h27m13.51460277s) [noise=true , address=145.79.227.39:4014 , piece_id=ZLJ7L4OXXXNMJP2NTDU5HNE7NDCFL6JNWLXB6KVZHHFZXF5AT4CA, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:37.047232213 (24h27m13.518442952s) [noise=true , address=145.79.227.29:4942 , piece_id=QOOXF6G2OGBNJLA64B2YNVVJKAZSP6HM5ZIEI4ZFJKWZUSQAR3XA, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:37.315831267 (24h27m13.787042006s) [noise=true , address=145.79.227.10:4401 , piece_id=TLP7B6MNA7W35UJQKW5SW6F43YXXMUFXOUKW4DKSOWOPIKLSSVUA, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:37.332403453 (24h27m13.803614192s) [noise=true , address=145.79.227.43:4157 , piece_id=VWS32XVNWH4R5PZDDV4FGP5ONZQZNLMU5YYB7QYS43GDJSDFX2OQ, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:37.332432619 (24h27m13.803643358s) [seg_pos={0 0}, upload=21507]: Segment reached optimal threshold of 13 pieces. Dec 4 17:22:37.332650384 (24h27m13.803861123s) [noise=true , address=145.79.227.9:4366 , piece_id=GDCC3W66FFB27QD5MRCG2MBJXZBWXRIWG7A4L3XMFTIRNN2NYPUQ, seg_pos={0 0}, upload=21507]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy): protocol: context canceled Dec 4 17:22:37.332660022 (24h27m13.803870751s) [noise=true , address=145.79.227.9:4366 , piece_id=GDCC3W66FFB27QD5MRCG2MBJXZBWXRIWG7A4L3XMFTIRNN2NYPUQ, seg_pos={0 0}, upload=21507]: Long tail cancellation due to optimal threshold Dec 4 17:22:37.332942179 (24h27m13.804152938s) [noise=true , address=145.79.227.43:4158 , piece_id=74VKW6UWNPLSDKXJ6AG6TTUSDBCW4K4BAGMFIDNIV55AIWRBVB2Q, seg_pos={0 0}, upload=21507]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vJZrSvPvwebw9V2tTbvkPEmLeLjryBUnq9TL6NwdPsHU9xcq2): protocol: expected piece hash; context canceled; EOF Dec 4 17:22:37.332954993 (24h27m13.804165732s) [noise=true , address=145.79.227.43:4158 , piece_id=74VKW6UWNPLSDKXJ6AG6TTUSDBCW4K4BAGMFIDNIV55AIWRBVB2Q, seg_pos={0 0}, upload=21507]: Long tail cancellation due to optimal threshold Dec 4 17:22:37.332976635 (24h27m13.804187384s) [seg_pos={0 0}, upload=21507]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 4 17:22:37.333114427 (24h27m13.804325166s) [upload=21507]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 4 17:22:37.333121821 (24h27m13.80433255s) [upload=21507]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 4 17:22:37.958438444 (24h27m14.429649173s) [upload=21507]: Done uploading segments... Dec 4 17:22:37.958456208 (24h27m14.429666937s) [upload=21507]: Upload finished. err: <nil> Duration: 5.818325864s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [............................X...............................] duration: 3.444779909s trace: https://jaeger.dev.storj.io/trace/1f3e769edbb82b7a FAIL (2025-12-04 17:27:06 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.444811369s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)