Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [................................X...........................] duration: 3.480841343s trace: https://jaeger.dev.storj.io/trace/1f3e769edbb82b7a FAIL (2025-12-04 17:24:21 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.480862414s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 5.790224515s trace: https://jaeger.dev.storj.io/trace/6d15aafc5ce7b199 FAIL (2025-12-04 17:24:20 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, Piece ID: FBD34B6US36TX6YHN5UEBKOWIVOOP4G7B6SSYKN6Q3J5YNE7OBHA) context deadline exceeded; piecestore: (Node ID: 12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9, Piece ID: ULLE3QB2QSCSYQWZ4OSBDJHC3X7AOHJGRWRA4DUXAMOYBDJRYWRQ) context deadline exceeded; piecestore: (Node ID: 1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, Piece ID: TLP7B6MNA7W35UJQKW5SW6F43YXXMUFXOUKW4DKSOWOPIKLSSVUA) context deadline exceeded; piecestore: (Node ID: 17o881pUMuG7K2dv83k7RkrJCrmM7kgUiWesgam4GEw21KVUn7, Piece ID: QOOXF6G2OGBNJLA64B2YNVVJKAZSP6HM5ZIEI4ZFJKWZUSQAR3XA) context deadline exceeded; piecestore: (Node ID: 1ZrzADk2mKG4R9tjxUKjQeJvb4WmfjLksGFyvstHRfYbBC7XNz, Piece ID: VWS32XVNWH4R5PZDDV4FGP5ONZQZNLMU5YYB7QYS43GDJSDFX2OQ) context deadline exceeded ======LOGS====== Dec 4 17:22:34.320746327 (24h27m10.791957066s) [upload=21507]: Starting upload Dec 4 17:22:34.320860715 (24h27m10.792071454s) [upload=21507]: Done starting upload Dec 4 17:22:34.321208968 (24h27m10.792419707s) [upload=21507]: Uploading segments... Dec 4 17:22:34.321325259 (24h27m10.792536018s) [upload=21507]: Got next segment. Inline: false Dec 4 17:22:34.321394982 (24h27m10.792605711s) [upload=21507]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 17:22:34.321401375 (24h27m10.792612114s) [upload=21507]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 17:22:34.827776318 (24h27m11.298987057s) [seg_pos={0 0}, upload=21507]: Begin upload segment... Dec 4 17:22:34.827838226 (24h27m11.299048966s) [seg_pos={0 0}, upload=21507]: Done begin upload segment. Dec 4 17:22:34.827877361 (24h27m11.29908811s) [noise=true , address=77.90.31.14:4661 , piece_id=D5L5WWE5DGEYKP5M2NM74EGSX5BLEP4DX5OBM2Z4Q2QTCSGA4WXQ, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.827887630 (24h27m11.299098359s) [noise=true , address=77.90.31.21:4878 , piece_id=GHU3SZUB5WAMW37CYCUFJ6HOPMXRNZGIJYDC4DJ3WOHV7YYWCVEA, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.828727861 (24h27m11.2999386s) [noise=true , address=145.79.227.29:4942 , piece_id=QOOXF6G2OGBNJLA64B2YNVVJKAZSP6HM5ZIEI4ZFJKWZUSQAR3XA, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.828821058 (24h27m11.300031807s) [noise=true , address=145.79.227.16:4910 , piece_id=36JF4CMQH5JO66BY5A53QNZQ6RIFTHZ6MD2QLBGHLHML5VBFGK7A, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.829248923 (24h27m11.300459652s) [noise=true , address=145.79.227.43:4157 , piece_id=VWS32XVNWH4R5PZDDV4FGP5ONZQZNLMU5YYB7QYS43GDJSDFX2OQ, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.829618296 (24h27m11.300829035s) [noise=true , address=145.79.227.38:4978 , piece_id=JLYEPL2WLUJIIUHI3FSOWIT5QG3ZQG3NNOZZ3TNMAOEWLZJNN6AA, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.830015262 (24h27m11.301226001s) [noise=true , address=77.90.31.19:4843 , piece_id=36QLYXSVN6L2FDKVX4WS7MRTK54XLB2JUTW2NKHCE7E6KZGTLUKQ, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.830278002 (24h27m11.301488741s) [noise=true , address=145.79.227.21:4475 , piece_id=YZRHRIA7WWLZKDX3BZ2FSUOKJ3MX7D7Q2GB6CHKVQKGXPQCUNC6A, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.830415063 (24h27m11.301625802s) [noise=true , address=145.79.227.8:4329 , piece_id=FBD34B6US36TX6YHN5UEBKOWIVOOP4G7B6SSYKN6Q3J5YNE7OBHA, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.830692902 (24h27m11.301903641s) [noise=true , address=145.79.227.9:4366 , piece_id=GDCC3W66FFB27QD5MRCG2MBJXZBWXRIWG7A4L3XMFTIRNN2NYPUQ, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.830811337 (24h27m11.302022076s) [noise=true , address=77.90.31.10:4520 , piece_id=DGMWUPIGVLYJ4IH2CXCHA3U7FHDJF7APG73XN6TUEWRBB3WZOURQ, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.831349433 (24h27m11.302560172s) [noise=true , address=77.90.31.18:4807 , piece_id=WJ3AOAEE3IG3T3QQRFSKTOJCYVST4263Y5UN6XVOVTSO5ALOVTCQ, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.831805731 (24h27m11.30301647s) [noise=true , address=145.79.227.43:4158 , piece_id=74VKW6UWNPLSDKXJ6AG6TTUSDBCW4K4BAGMFIDNIV55AIWRBVB2Q, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.832172379 (24h27m11.303383118s) [noise=true , address=145.79.227.39:4014 , piece_id=ZLJ7L4OXXXNMJP2NTDU5HNE7NDCFL6JNWLXB6KVZHHFZXF5AT4CA, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.833230595 (24h27m11.304441334s) [noise=true , address=145.79.227.34:4905 , piece_id=ULLE3QB2QSCSYQWZ4OSBDJHC3X7AOHJGRWRA4DUXAMOYBDJRYWRQ, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.833972006 (24h27m11.305182755s) [noise=true , address=145.79.227.10:4401 , piece_id=TLP7B6MNA7W35UJQKW5SW6F43YXXMUFXOUKW4DKSOWOPIKLSSVUA, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:35.930150729 (24h27m12.401361488s) [noise=true , address=77.90.31.19:4843 , piece_id=36QLYXSVN6L2FDKVX4WS7MRTK54XLB2JUTW2NKHCE7E6KZGTLUKQ, seg_pos={0 0}, upload=21507]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: storage node overloaded, request limit: 2000 Dec 4 17:22:36.479057771 (24h27m12.9502685s) [upload=21507]: Next returned nil segment Dec 4 17:22:36.479122765 (24h27m12.950333494s) [upload=21507]: Waiting for error group managing segments... Dec 4 17:22:36.876009453 (24h27m13.347220192s) [noise=true , address=77.90.31.21:4878 , piece_id=GHU3SZUB5WAMW37CYCUFJ6HOPMXRNZGIJYDC4DJ3WOHV7YYWCVEA, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:36.891015988 (24h27m13.362226717s) [noise=true , address=77.90.31.14:4661 , piece_id=D5L5WWE5DGEYKP5M2NM74EGSX5BLEP4DX5OBM2Z4Q2QTCSGA4WXQ, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:36.943731919 (24h27m13.414942658s) [noise=true , address=77.90.31.10:4520 , piece_id=DGMWUPIGVLYJ4IH2CXCHA3U7FHDJF7APG73XN6TUEWRBB3WZOURQ, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:36.981669560 (24h27m13.452880289s) [noise=true , address=145.79.227.16:4910 , piece_id=36JF4CMQH5JO66BY5A53QNZQ6RIFTHZ6MD2QLBGHLHML5VBFGK7A, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:36.981785620 (24h27m13.452996359s) [noise=true , address=145.79.227.21:4475 , piece_id=YZRHRIA7WWLZKDX3BZ2FSUOKJ3MX7D7Q2GB6CHKVQKGXPQCUNC6A, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:36.984955717 (24h27m13.456166456s) [noise=true , address=77.90.31.18:4807 , piece_id=WJ3AOAEE3IG3T3QQRFSKTOJCYVST4263Y5UN6XVOVTSO5ALOVTCQ, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:36.994452571 (24h27m13.4656633s) [noise=true , address=145.79.227.34:4905 , piece_id=ULLE3QB2QSCSYQWZ4OSBDJHC3X7AOHJGRWRA4DUXAMOYBDJRYWRQ, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:37.011055396 (24h27m13.482266125s) [noise=true , address=145.79.227.38:4978 , piece_id=JLYEPL2WLUJIIUHI3FSOWIT5QG3ZQG3NNOZZ3TNMAOEWLZJNN6AA, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:37.021499513 (24h27m13.492710252s) [noise=true , address=145.79.227.8:4329 , piece_id=FBD34B6US36TX6YHN5UEBKOWIVOOP4G7B6SSYKN6Q3J5YNE7OBHA, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:37.043392031 (24h27m13.51460277s) [noise=true , address=145.79.227.39:4014 , piece_id=ZLJ7L4OXXXNMJP2NTDU5HNE7NDCFL6JNWLXB6KVZHHFZXF5AT4CA, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:37.047232213 (24h27m13.518442952s) [noise=true , address=145.79.227.29:4942 , piece_id=QOOXF6G2OGBNJLA64B2YNVVJKAZSP6HM5ZIEI4ZFJKWZUSQAR3XA, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:37.315831267 (24h27m13.787042006s) [noise=true , address=145.79.227.10:4401 , piece_id=TLP7B6MNA7W35UJQKW5SW6F43YXXMUFXOUKW4DKSOWOPIKLSSVUA, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:37.332403453 (24h27m13.803614192s) [noise=true , address=145.79.227.43:4157 , piece_id=VWS32XVNWH4R5PZDDV4FGP5ONZQZNLMU5YYB7QYS43GDJSDFX2OQ, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:37.332432619 (24h27m13.803643358s) [seg_pos={0 0}, upload=21507]: Segment reached optimal threshold of 13 pieces. Dec 4 17:22:37.332650384 (24h27m13.803861123s) [noise=true , address=145.79.227.9:4366 , piece_id=GDCC3W66FFB27QD5MRCG2MBJXZBWXRIWG7A4L3XMFTIRNN2NYPUQ, seg_pos={0 0}, upload=21507]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy): protocol: context canceled Dec 4 17:22:37.332660022 (24h27m13.803870751s) [noise=true , address=145.79.227.9:4366 , piece_id=GDCC3W66FFB27QD5MRCG2MBJXZBWXRIWG7A4L3XMFTIRNN2NYPUQ, seg_pos={0 0}, upload=21507]: Long tail cancellation due to optimal threshold Dec 4 17:22:37.332942179 (24h27m13.804152938s) [noise=true , address=145.79.227.43:4158 , piece_id=74VKW6UWNPLSDKXJ6AG6TTUSDBCW4K4BAGMFIDNIV55AIWRBVB2Q, seg_pos={0 0}, upload=21507]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vJZrSvPvwebw9V2tTbvkPEmLeLjryBUnq9TL6NwdPsHU9xcq2): protocol: expected piece hash; context canceled; EOF Dec 4 17:22:37.332954993 (24h27m13.804165732s) [noise=true , address=145.79.227.43:4158 , piece_id=74VKW6UWNPLSDKXJ6AG6TTUSDBCW4K4BAGMFIDNIV55AIWRBVB2Q, seg_pos={0 0}, upload=21507]: Long tail cancellation due to optimal threshold Dec 4 17:22:37.332976635 (24h27m13.804187384s) [seg_pos={0 0}, upload=21507]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 4 17:22:37.333114427 (24h27m13.804325166s) [upload=21507]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 4 17:22:37.333121821 (24h27m13.80433255s) [upload=21507]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 4 17:22:37.958438444 (24h27m14.429649173s) [upload=21507]: Done uploading segments... Dec 4 17:22:37.958456208 (24h27m14.429666937s) [upload=21507]: Upload finished. err: <nil> Duration: 5.790255906s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.................................X..........................] duration: 3.495488986s trace: https://jaeger.dev.storj.io/trace/1f3e769edbb82b7a FAIL (2025-12-04 17:23:39 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.495531769s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 5.910251471s trace: https://jaeger.dev.storj.io/trace/6d15aafc5ce7b199 FAIL (2025-12-04 17:23:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, Piece ID: FBD34B6US36TX6YHN5UEBKOWIVOOP4G7B6SSYKN6Q3J5YNE7OBHA) context deadline exceeded; piecestore: (Node ID: 12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9, Piece ID: ULLE3QB2QSCSYQWZ4OSBDJHC3X7AOHJGRWRA4DUXAMOYBDJRYWRQ) context deadline exceeded; piecestore: (Node ID: 1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, Piece ID: TLP7B6MNA7W35UJQKW5SW6F43YXXMUFXOUKW4DKSOWOPIKLSSVUA) context deadline exceeded; piecestore: (Node ID: 17o881pUMuG7K2dv83k7RkrJCrmM7kgUiWesgam4GEw21KVUn7, Piece ID: QOOXF6G2OGBNJLA64B2YNVVJKAZSP6HM5ZIEI4ZFJKWZUSQAR3XA) context deadline exceeded; piecestore: (Node ID: 1ZrzADk2mKG4R9tjxUKjQeJvb4WmfjLksGFyvstHRfYbBC7XNz, Piece ID: VWS32XVNWH4R5PZDDV4FGP5ONZQZNLMU5YYB7QYS43GDJSDFX2OQ) context deadline exceeded ======LOGS====== Dec 4 17:22:34.320746327 (24h27m10.791957066s) [upload=21507]: Starting upload Dec 4 17:22:34.320860715 (24h27m10.792071454s) [upload=21507]: Done starting upload Dec 4 17:22:34.321208968 (24h27m10.792419707s) [upload=21507]: Uploading segments... Dec 4 17:22:34.321325259 (24h27m10.792536018s) [upload=21507]: Got next segment. Inline: false Dec 4 17:22:34.321394982 (24h27m10.792605711s) [upload=21507]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 17:22:34.321401375 (24h27m10.792612114s) [upload=21507]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 17:22:34.827776318 (24h27m11.298987057s) [seg_pos={0 0}, upload=21507]: Begin upload segment... Dec 4 17:22:34.827838226 (24h27m11.299048966s) [seg_pos={0 0}, upload=21507]: Done begin upload segment. Dec 4 17:22:34.827877361 (24h27m11.29908811s) [noise=true , address=77.90.31.14:4661 , piece_id=D5L5WWE5DGEYKP5M2NM74EGSX5BLEP4DX5OBM2Z4Q2QTCSGA4WXQ, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.827887630 (24h27m11.299098359s) [noise=true , address=77.90.31.21:4878 , piece_id=GHU3SZUB5WAMW37CYCUFJ6HOPMXRNZGIJYDC4DJ3WOHV7YYWCVEA, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.828727861 (24h27m11.2999386s) [noise=true , address=145.79.227.29:4942 , piece_id=QOOXF6G2OGBNJLA64B2YNVVJKAZSP6HM5ZIEI4ZFJKWZUSQAR3XA, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.828821058 (24h27m11.300031807s) [noise=true , address=145.79.227.16:4910 , piece_id=36JF4CMQH5JO66BY5A53QNZQ6RIFTHZ6MD2QLBGHLHML5VBFGK7A, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.829248923 (24h27m11.300459652s) [noise=true , address=145.79.227.43:4157 , piece_id=VWS32XVNWH4R5PZDDV4FGP5ONZQZNLMU5YYB7QYS43GDJSDFX2OQ, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.829618296 (24h27m11.300829035s) [noise=true , address=145.79.227.38:4978 , piece_id=JLYEPL2WLUJIIUHI3FSOWIT5QG3ZQG3NNOZZ3TNMAOEWLZJNN6AA, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.830015262 (24h27m11.301226001s) [noise=true , address=77.90.31.19:4843 , piece_id=36QLYXSVN6L2FDKVX4WS7MRTK54XLB2JUTW2NKHCE7E6KZGTLUKQ, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.830278002 (24h27m11.301488741s) [noise=true , address=145.79.227.21:4475 , piece_id=YZRHRIA7WWLZKDX3BZ2FSUOKJ3MX7D7Q2GB6CHKVQKGXPQCUNC6A, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.830415063 (24h27m11.301625802s) [noise=true , address=145.79.227.8:4329 , piece_id=FBD34B6US36TX6YHN5UEBKOWIVOOP4G7B6SSYKN6Q3J5YNE7OBHA, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.830692902 (24h27m11.301903641s) [noise=true , address=145.79.227.9:4366 , piece_id=GDCC3W66FFB27QD5MRCG2MBJXZBWXRIWG7A4L3XMFTIRNN2NYPUQ, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.830811337 (24h27m11.302022076s) [noise=true , address=77.90.31.10:4520 , piece_id=DGMWUPIGVLYJ4IH2CXCHA3U7FHDJF7APG73XN6TUEWRBB3WZOURQ, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.831349433 (24h27m11.302560172s) [noise=true , address=77.90.31.18:4807 , piece_id=WJ3AOAEE3IG3T3QQRFSKTOJCYVST4263Y5UN6XVOVTSO5ALOVTCQ, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.831805731 (24h27m11.30301647s) [noise=true , address=145.79.227.43:4158 , piece_id=74VKW6UWNPLSDKXJ6AG6TTUSDBCW4K4BAGMFIDNIV55AIWRBVB2Q, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.832172379 (24h27m11.303383118s) [noise=true , address=145.79.227.39:4014 , piece_id=ZLJ7L4OXXXNMJP2NTDU5HNE7NDCFL6JNWLXB6KVZHHFZXF5AT4CA, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.833230595 (24h27m11.304441334s) [noise=true , address=145.79.227.34:4905 , piece_id=ULLE3QB2QSCSYQWZ4OSBDJHC3X7AOHJGRWRA4DUXAMOYBDJRYWRQ, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.833972006 (24h27m11.305182755s) [noise=true , address=145.79.227.10:4401 , piece_id=TLP7B6MNA7W35UJQKW5SW6F43YXXMUFXOUKW4DKSOWOPIKLSSVUA, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:35.930150729 (24h27m12.401361488s) [noise=true , address=77.90.31.19:4843 , piece_id=36QLYXSVN6L2FDKVX4WS7MRTK54XLB2JUTW2NKHCE7E6KZGTLUKQ, seg_pos={0 0}, upload=21507]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: storage node overloaded, request limit: 2000 Dec 4 17:22:36.479057771 (24h27m12.9502685s) [upload=21507]: Next returned nil segment Dec 4 17:22:36.479122765 (24h27m12.950333494s) [upload=21507]: Waiting for error group managing segments... Dec 4 17:22:36.876009453 (24h27m13.347220192s) [noise=true , address=77.90.31.21:4878 , piece_id=GHU3SZUB5WAMW37CYCUFJ6HOPMXRNZGIJYDC4DJ3WOHV7YYWCVEA, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:36.891015988 (24h27m13.362226717s) [noise=true , address=77.90.31.14:4661 , piece_id=D5L5WWE5DGEYKP5M2NM74EGSX5BLEP4DX5OBM2Z4Q2QTCSGA4WXQ, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:36.943731919 (24h27m13.414942658s) [noise=true , address=77.90.31.10:4520 , piece_id=DGMWUPIGVLYJ4IH2CXCHA3U7FHDJF7APG73XN6TUEWRBB3WZOURQ, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:36.981669560 (24h27m13.452880289s) [noise=true , address=145.79.227.16:4910 , piece_id=36JF4CMQH5JO66BY5A53QNZQ6RIFTHZ6MD2QLBGHLHML5VBFGK7A, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:36.981785620 (24h27m13.452996359s) [noise=true , address=145.79.227.21:4475 , piece_id=YZRHRIA7WWLZKDX3BZ2FSUOKJ3MX7D7Q2GB6CHKVQKGXPQCUNC6A, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:36.984955717 (24h27m13.456166456s) [noise=true , address=77.90.31.18:4807 , piece_id=WJ3AOAEE3IG3T3QQRFSKTOJCYVST4263Y5UN6XVOVTSO5ALOVTCQ, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:36.994452571 (24h27m13.4656633s) [noise=true , address=145.79.227.34:4905 , piece_id=ULLE3QB2QSCSYQWZ4OSBDJHC3X7AOHJGRWRA4DUXAMOYBDJRYWRQ, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:37.011055396 (24h27m13.482266125s) [noise=true , address=145.79.227.38:4978 , piece_id=JLYEPL2WLUJIIUHI3FSOWIT5QG3ZQG3NNOZZ3TNMAOEWLZJNN6AA, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:37.021499513 (24h27m13.492710252s) [noise=true , address=145.79.227.8:4329 , piece_id=FBD34B6US36TX6YHN5UEBKOWIVOOP4G7B6SSYKN6Q3J5YNE7OBHA, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:37.043392031 (24h27m13.51460277s) [noise=true , address=145.79.227.39:4014 , piece_id=ZLJ7L4OXXXNMJP2NTDU5HNE7NDCFL6JNWLXB6KVZHHFZXF5AT4CA, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:37.047232213 (24h27m13.518442952s) [noise=true , address=145.79.227.29:4942 , piece_id=QOOXF6G2OGBNJLA64B2YNVVJKAZSP6HM5ZIEI4ZFJKWZUSQAR3XA, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:37.315831267 (24h27m13.787042006s) [noise=true , address=145.79.227.10:4401 , piece_id=TLP7B6MNA7W35UJQKW5SW6F43YXXMUFXOUKW4DKSOWOPIKLSSVUA, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:37.332403453 (24h27m13.803614192s) [noise=true , address=145.79.227.43:4157 , piece_id=VWS32XVNWH4R5PZDDV4FGP5ONZQZNLMU5YYB7QYS43GDJSDFX2OQ, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:37.332432619 (24h27m13.803643358s) [seg_pos={0 0}, upload=21507]: Segment reached optimal threshold of 13 pieces. Dec 4 17:22:37.332650384 (24h27m13.803861123s) [noise=true , address=145.79.227.9:4366 , piece_id=GDCC3W66FFB27QD5MRCG2MBJXZBWXRIWG7A4L3XMFTIRNN2NYPUQ, seg_pos={0 0}, upload=21507]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy): protocol: context canceled Dec 4 17:22:37.332660022 (24h27m13.803870751s) [noise=true , address=145.79.227.9:4366 , piece_id=GDCC3W66FFB27QD5MRCG2MBJXZBWXRIWG7A4L3XMFTIRNN2NYPUQ, seg_pos={0 0}, upload=21507]: Long tail cancellation due to optimal threshold Dec 4 17:22:37.332942179 (24h27m13.804152938s) [noise=true , address=145.79.227.43:4158 , piece_id=74VKW6UWNPLSDKXJ6AG6TTUSDBCW4K4BAGMFIDNIV55AIWRBVB2Q, seg_pos={0 0}, upload=21507]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vJZrSvPvwebw9V2tTbvkPEmLeLjryBUnq9TL6NwdPsHU9xcq2): protocol: expected piece hash; context canceled; EOF Dec 4 17:22:37.332954993 (24h27m13.804165732s) [noise=true , address=145.79.227.43:4158 , piece_id=74VKW6UWNPLSDKXJ6AG6TTUSDBCW4K4BAGMFIDNIV55AIWRBVB2Q, seg_pos={0 0}, upload=21507]: Long tail cancellation due to optimal threshold Dec 4 17:22:37.332976635 (24h27m13.804187384s) [seg_pos={0 0}, upload=21507]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 4 17:22:37.333114427 (24h27m13.804325166s) [upload=21507]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 4 17:22:37.333121821 (24h27m13.80433255s) [upload=21507]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 4 17:22:37.958438444 (24h27m14.429649173s) [upload=21507]: Done uploading segments... Dec 4 17:22:37.958456208 (24h27m14.429666937s) [upload=21507]: Upload finished. err: <nil> Duration: 5.910281749s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.003173831s trace: https://jaeger.dev.storj.io/trace/6d15aafc5ce7b199 FAIL (2025-12-04 17:23:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, Piece ID: FBD34B6US36TX6YHN5UEBKOWIVOOP4G7B6SSYKN6Q3J5YNE7OBHA) context deadline exceeded; piecestore: (Node ID: 12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9, Piece ID: ULLE3QB2QSCSYQWZ4OSBDJHC3X7AOHJGRWRA4DUXAMOYBDJRYWRQ) context deadline exceeded; piecestore: (Node ID: 1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, Piece ID: TLP7B6MNA7W35UJQKW5SW6F43YXXMUFXOUKW4DKSOWOPIKLSSVUA) context deadline exceeded; piecestore: (Node ID: 17o881pUMuG7K2dv83k7RkrJCrmM7kgUiWesgam4GEw21KVUn7, Piece ID: QOOXF6G2OGBNJLA64B2YNVVJKAZSP6HM5ZIEI4ZFJKWZUSQAR3XA) context deadline exceeded; piecestore: (Node ID: 1ZrzADk2mKG4R9tjxUKjQeJvb4WmfjLksGFyvstHRfYbBC7XNz, Piece ID: VWS32XVNWH4R5PZDDV4FGP5ONZQZNLMU5YYB7QYS43GDJSDFX2OQ) context deadline exceeded ======LOGS====== Dec 4 17:22:34.320746327 (24h27m10.791957066s) [upload=21507]: Starting upload Dec 4 17:22:34.320860715 (24h27m10.792071454s) [upload=21507]: Done starting upload Dec 4 17:22:34.321208968 (24h27m10.792419707s) [upload=21507]: Uploading segments... Dec 4 17:22:34.321325259 (24h27m10.792536018s) [upload=21507]: Got next segment. Inline: false Dec 4 17:22:34.321394982 (24h27m10.792605711s) [upload=21507]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 17:22:34.321401375 (24h27m10.792612114s) [upload=21507]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 17:22:34.827776318 (24h27m11.298987057s) [seg_pos={0 0}, upload=21507]: Begin upload segment... Dec 4 17:22:34.827838226 (24h27m11.299048966s) [seg_pos={0 0}, upload=21507]: Done begin upload segment. Dec 4 17:22:34.827877361 (24h27m11.29908811s) [noise=true , address=77.90.31.14:4661 , piece_id=D5L5WWE5DGEYKP5M2NM74EGSX5BLEP4DX5OBM2Z4Q2QTCSGA4WXQ, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.827887630 (24h27m11.299098359s) [noise=true , address=77.90.31.21:4878 , piece_id=GHU3SZUB5WAMW37CYCUFJ6HOPMXRNZGIJYDC4DJ3WOHV7YYWCVEA, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.828727861 (24h27m11.2999386s) [noise=true , address=145.79.227.29:4942 , piece_id=QOOXF6G2OGBNJLA64B2YNVVJKAZSP6HM5ZIEI4ZFJKWZUSQAR3XA, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.828821058 (24h27m11.300031807s) [noise=true , address=145.79.227.16:4910 , piece_id=36JF4CMQH5JO66BY5A53QNZQ6RIFTHZ6MD2QLBGHLHML5VBFGK7A, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.829248923 (24h27m11.300459652s) [noise=true , address=145.79.227.43:4157 , piece_id=VWS32XVNWH4R5PZDDV4FGP5ONZQZNLMU5YYB7QYS43GDJSDFX2OQ, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.829618296 (24h27m11.300829035s) [noise=true , address=145.79.227.38:4978 , piece_id=JLYEPL2WLUJIIUHI3FSOWIT5QG3ZQG3NNOZZ3TNMAOEWLZJNN6AA, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.830015262 (24h27m11.301226001s) [noise=true , address=77.90.31.19:4843 , piece_id=36QLYXSVN6L2FDKVX4WS7MRTK54XLB2JUTW2NKHCE7E6KZGTLUKQ, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.830278002 (24h27m11.301488741s) [noise=true , address=145.79.227.21:4475 , piece_id=YZRHRIA7WWLZKDX3BZ2FSUOKJ3MX7D7Q2GB6CHKVQKGXPQCUNC6A, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.830415063 (24h27m11.301625802s) [noise=true , address=145.79.227.8:4329 , piece_id=FBD34B6US36TX6YHN5UEBKOWIVOOP4G7B6SSYKN6Q3J5YNE7OBHA, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.830692902 (24h27m11.301903641s) [noise=true , address=145.79.227.9:4366 , piece_id=GDCC3W66FFB27QD5MRCG2MBJXZBWXRIWG7A4L3XMFTIRNN2NYPUQ, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.830811337 (24h27m11.302022076s) [noise=true , address=77.90.31.10:4520 , piece_id=DGMWUPIGVLYJ4IH2CXCHA3U7FHDJF7APG73XN6TUEWRBB3WZOURQ, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.831349433 (24h27m11.302560172s) [noise=true , address=77.90.31.18:4807 , piece_id=WJ3AOAEE3IG3T3QQRFSKTOJCYVST4263Y5UN6XVOVTSO5ALOVTCQ, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.831805731 (24h27m11.30301647s) [noise=true , address=145.79.227.43:4158 , piece_id=74VKW6UWNPLSDKXJ6AG6TTUSDBCW4K4BAGMFIDNIV55AIWRBVB2Q, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.832172379 (24h27m11.303383118s) [noise=true , address=145.79.227.39:4014 , piece_id=ZLJ7L4OXXXNMJP2NTDU5HNE7NDCFL6JNWLXB6KVZHHFZXF5AT4CA, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.833230595 (24h27m11.304441334s) [noise=true , address=145.79.227.34:4905 , piece_id=ULLE3QB2QSCSYQWZ4OSBDJHC3X7AOHJGRWRA4DUXAMOYBDJRYWRQ, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:34.833972006 (24h27m11.305182755s) [noise=true , address=145.79.227.10:4401 , piece_id=TLP7B6MNA7W35UJQKW5SW6F43YXXMUFXOUKW4DKSOWOPIKLSSVUA, seg_pos={0 0}, upload=21507]: Uploading piece... Dec 4 17:22:35.930150729 (24h27m12.401361488s) [noise=true , address=77.90.31.19:4843 , piece_id=36QLYXSVN6L2FDKVX4WS7MRTK54XLB2JUTW2NKHCE7E6KZGTLUKQ, seg_pos={0 0}, upload=21507]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: storage node overloaded, request limit: 2000 Dec 4 17:22:36.479057771 (24h27m12.9502685s) [upload=21507]: Next returned nil segment Dec 4 17:22:36.479122765 (24h27m12.950333494s) [upload=21507]: Waiting for error group managing segments... Dec 4 17:22:36.876009453 (24h27m13.347220192s) [noise=true , address=77.90.31.21:4878 , piece_id=GHU3SZUB5WAMW37CYCUFJ6HOPMXRNZGIJYDC4DJ3WOHV7YYWCVEA, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:36.891015988 (24h27m13.362226717s) [noise=true , address=77.90.31.14:4661 , piece_id=D5L5WWE5DGEYKP5M2NM74EGSX5BLEP4DX5OBM2Z4Q2QTCSGA4WXQ, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:36.943731919 (24h27m13.414942658s) [noise=true , address=77.90.31.10:4520 , piece_id=DGMWUPIGVLYJ4IH2CXCHA3U7FHDJF7APG73XN6TUEWRBB3WZOURQ, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:36.981669560 (24h27m13.452880289s) [noise=true , address=145.79.227.16:4910 , piece_id=36JF4CMQH5JO66BY5A53QNZQ6RIFTHZ6MD2QLBGHLHML5VBFGK7A, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:36.981785620 (24h27m13.452996359s) [noise=true , address=145.79.227.21:4475 , piece_id=YZRHRIA7WWLZKDX3BZ2FSUOKJ3MX7D7Q2GB6CHKVQKGXPQCUNC6A, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:36.984955717 (24h27m13.456166456s) [noise=true , address=77.90.31.18:4807 , piece_id=WJ3AOAEE3IG3T3QQRFSKTOJCYVST4263Y5UN6XVOVTSO5ALOVTCQ, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:36.994452571 (24h27m13.4656633s) [noise=true , address=145.79.227.34:4905 , piece_id=ULLE3QB2QSCSYQWZ4OSBDJHC3X7AOHJGRWRA4DUXAMOYBDJRYWRQ, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:37.011055396 (24h27m13.482266125s) [noise=true , address=145.79.227.38:4978 , piece_id=JLYEPL2WLUJIIUHI3FSOWIT5QG3ZQG3NNOZZ3TNMAOEWLZJNN6AA, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:37.021499513 (24h27m13.492710252s) [noise=true , address=145.79.227.8:4329 , piece_id=FBD34B6US36TX6YHN5UEBKOWIVOOP4G7B6SSYKN6Q3J5YNE7OBHA, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:37.043392031 (24h27m13.51460277s) [noise=true , address=145.79.227.39:4014 , piece_id=ZLJ7L4OXXXNMJP2NTDU5HNE7NDCFL6JNWLXB6KVZHHFZXF5AT4CA, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:37.047232213 (24h27m13.518442952s) [noise=true , address=145.79.227.29:4942 , piece_id=QOOXF6G2OGBNJLA64B2YNVVJKAZSP6HM5ZIEI4ZFJKWZUSQAR3XA, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:37.315831267 (24h27m13.787042006s) [noise=true , address=145.79.227.10:4401 , piece_id=TLP7B6MNA7W35UJQKW5SW6F43YXXMUFXOUKW4DKSOWOPIKLSSVUA, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:37.332403453 (24h27m13.803614192s) [noise=true , address=145.79.227.43:4157 , piece_id=VWS32XVNWH4R5PZDDV4FGP5ONZQZNLMU5YYB7QYS43GDJSDFX2OQ, seg_pos={0 0}, upload=21507]: Done uploading piece. err: <nil> Dec 4 17:22:37.332432619 (24h27m13.803643358s) [seg_pos={0 0}, upload=21507]: Segment reached optimal threshold of 13 pieces. Dec 4 17:22:37.332650384 (24h27m13.803861123s) [noise=true , address=145.79.227.9:4366 , piece_id=GDCC3W66FFB27QD5MRCG2MBJXZBWXRIWG7A4L3XMFTIRNN2NYPUQ, seg_pos={0 0}, upload=21507]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy): protocol: context canceled Dec 4 17:22:37.332660022 (24h27m13.803870751s) [noise=true , address=145.79.227.9:4366 , piece_id=GDCC3W66FFB27QD5MRCG2MBJXZBWXRIWG7A4L3XMFTIRNN2NYPUQ, seg_pos={0 0}, upload=21507]: Long tail cancellation due to optimal threshold Dec 4 17:22:37.332942179 (24h27m13.804152938s) [noise=true , address=145.79.227.43:4158 , piece_id=74VKW6UWNPLSDKXJ6AG6TTUSDBCW4K4BAGMFIDNIV55AIWRBVB2Q, seg_pos={0 0}, upload=21507]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vJZrSvPvwebw9V2tTbvkPEmLeLjryBUnq9TL6NwdPsHU9xcq2): protocol: expected piece hash; context canceled; EOF Dec 4 17:22:37.332954993 (24h27m13.804165732s) [noise=true , address=145.79.227.43:4158 , piece_id=74VKW6UWNPLSDKXJ6AG6TTUSDBCW4K4BAGMFIDNIV55AIWRBVB2Q, seg_pos={0 0}, upload=21507]: Long tail cancellation due to optimal threshold Dec 4 17:22:37.332976635 (24h27m13.804187384s) [seg_pos={0 0}, upload=21507]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 4 17:22:37.333114427 (24h27m13.804325166s) [upload=21507]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 4 17:22:37.333121821 (24h27m13.80433255s) [upload=21507]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 4 17:22:37.958438444 (24h27m14.429649173s) [upload=21507]: Done uploading segments... Dec 4 17:22:37.958456208 (24h27m14.429666937s) [upload=21507]: Upload finished. err: <nil> Duration: 30.003198809s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..................................X.........................] duration: 3.412315336s trace: https://jaeger.dev.storj.io/trace/1f3e769edbb82b7a FAIL (2025-12-04 17:23:00 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.412391932s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...................................X........................] duration: 3.473804994s trace: https://jaeger.dev.storj.io/trace/1f3e769edbb82b7a FAIL (2025-12-04 17:22:23 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.47387069s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [....................................X.......................] duration: 3.56610893s trace: https://jaeger.dev.storj.io/trace/1f3e769edbb82b7a FAIL (2025-12-04 17:21:58 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.566139719s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.....................................X......................] duration: 3.484787999s trace: https://jaeger.dev.storj.io/trace/1f3e769edbb82b7a FAIL (2025-12-04 17:21:43 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.484821513s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [......................................X.....................] duration: 3.603523912s trace: https://jaeger.dev.storj.io/trace/1f3e769edbb82b7a FAIL (2025-12-04 17:20:54 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.603607331s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.......................................X....................] duration: 3.420779091s trace: https://jaeger.dev.storj.io/trace/1f3e769edbb82b7a FAIL (2025-12-04 17:20:14 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.420808125s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [........................................X...................] duration: 3.393833442s trace: https://jaeger.dev.storj.io/trace/1f3e769edbb82b7a FAIL (2025-12-04 17:19:43 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.39385823s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.........................................X..................] duration: 3.525629996s trace: https://jaeger.dev.storj.io/trace/1f3e769edbb82b7a FAIL (2025-12-04 17:19:10 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.525664141s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..........................................X.................] duration: 3.504183701s trace: https://jaeger.dev.storj.io/trace/1f3e769edbb82b7a FAIL (2025-12-04 17:18:47 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.504209169s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...........................................X................] duration: 3.464283594s trace: https://jaeger.dev.storj.io/trace/1f3e769edbb82b7a FAIL (2025-12-04 17:18:15 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.464370038s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [............................................X...............] duration: 3.513762418s trace: https://jaeger.dev.storj.io/trace/1f3e769edbb82b7a FAIL (2025-12-04 17:17:55 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.513818896s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.............................................X..............] duration: 3.554750702s trace: https://jaeger.dev.storj.io/trace/1f3e769edbb82b7a FAIL (2025-12-04 17:17:29 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.554779526s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..............................................X.............] duration: 3.573367981s trace: https://jaeger.dev.storj.io/trace/1f3e769edbb82b7a FAIL (2025-12-04 17:16:33 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.573420339s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...............................................X............] duration: 3.50312812s trace: https://jaeger.dev.storj.io/trace/1f3e769edbb82b7a FAIL (2025-12-04 17:15:27 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.50315979s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [................................................X...........] duration: 3.747870466s trace: https://jaeger.dev.storj.io/trace/1f3e769edbb82b7a FAIL (2025-12-04 17:14:38 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.747889512s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.................................................X..........] duration: 3.654680455s trace: https://jaeger.dev.storj.io/trace/1f3e769edbb82b7a FAIL (2025-12-04 17:14:04 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.654713367s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..................................................X.........] duration: 3.696976051s trace: https://jaeger.dev.storj.io/trace/1f3e769edbb82b7a FAIL (2025-12-04 17:13:04 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.697006869s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...................................................X........] duration: 3.875934966s trace: https://jaeger.dev.storj.io/trace/1f3e769edbb82b7a FAIL (2025-12-04 17:12:25 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.875964462s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [....................................................X.......] duration: 3.520844305s trace: https://jaeger.dev.storj.io/trace/1f3e769edbb82b7a FAIL (2025-12-04 17:11:25 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.520869363s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.....................................................X......] duration: 4.533434563s trace: https://jaeger.dev.storj.io/trace/1f3e769edbb82b7a FAIL (2025-12-04 17:10:18 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.533466693s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [......................................................X.....] duration: 3.722926037s trace: https://jaeger.dev.storj.io/trace/1f3e769edbb82b7a FAIL (2025-12-04 17:09:14 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.722955013s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.......................................................X....] duration: 3.899833854s trace: https://jaeger.dev.storj.io/trace/1f3e769edbb82b7a FAIL (2025-12-04 17:08:16 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.899884029s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [........................................................X...] duration: 3.555607877s trace: https://jaeger.dev.storj.io/trace/1f3e769edbb82b7a FAIL (2025-12-04 17:07:35 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.555631642s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.........................................................X..] duration: 3.691824294s trace: https://jaeger.dev.storj.io/trace/1f3e769edbb82b7a FAIL (2025-12-04 17:06:37 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.691852117s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..........................................................X.] duration: 4.55630801s trace: https://jaeger.dev.storj.io/trace/1f3e769edbb82b7a FAIL (2025-12-04 17:05:59 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.556331947s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...........................................................X] duration: 5.001191685s trace: https://jaeger.dev.storj.io/trace/1f3e769edbb82b7a FAIL (2025-12-04 17:05:37 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001214057s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 5.664247549s trace: https://jaeger.dev.storj.io/trace/1729cf59b626c72a FAIL (2025-12-04 16:17:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw, Piece ID: XNS7EVNDDK3XQVIYL5RPPIJPPTFNSTVJFWF6Q55EL3SRRAQTRYEQ) context deadline exceeded; piecestore: (Node ID: 12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, Piece ID: HYDMGVL463LIAO4N7JRFMFZSZ566C7DHHEMMVCPHTDUTEQYESKFQ) context deadline exceeded; piecestore: (Node ID: 1rL2DnjSgHP36Md3bCgy5padqTAhv3iQCeURbLMuuuMBxtUNZH, Piece ID: EQILGZ4H4R254DE7UW43GYCETN77D5FIRK4SE4Y36PPPSOKBSX5A) context deadline exceeded; piecestore: (Node ID: 12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n, Piece ID: RLDHK2A262X6IZQWP6DK2IPWBTMXYZ4YQQ7SG4IJRSO3KDPBUJYA) context deadline exceeded ======LOGS====== Dec 4 15:38:01.034643304 (22h42m37.505854043s) [upload=19952]: Starting upload Dec 4 15:38:01.034798069 (22h42m37.506008818s) [upload=19952]: Done starting upload Dec 4 15:38:01.034989493 (22h42m37.506200232s) [upload=19952]: Uploading segments... Dec 4 15:38:01.035032334 (22h42m37.506243073s) [upload=19952]: Got next segment. Inline: false Dec 4 15:38:01.035060508 (22h42m37.506271237s) [upload=19952]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 15:38:01.035064726 (22h42m37.506275455s) [upload=19952]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 15:38:01.301722830 (22h42m37.772933559s) [seg_pos={0 0}, upload=19952]: Begin upload segment... Dec 4 15:38:01.301843599 (22h42m37.773054338s) [seg_pos={0 0}, upload=19952]: Done begin upload segment. Dec 4 15:38:01.301883024 (22h42m37.773093753s) [noise=true , address=145.79.227.40:4051 , piece_id=RLDHK2A262X6IZQWP6DK2IPWBTMXYZ4YQQ7SG4IJRSO3KDPBUJYA, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.301945323 (22h42m37.773156092s) [noise=true , address=145.79.227.42:4121 , piece_id=USB7H5RUYH4JMFQQRO5M3BSKB4NETHDTSGV5CJRFOT76D32V6DHQ, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.302924858 (22h42m37.774135597s) [noise=true , address=145.79.227.23:4547 , piece_id=AGTUJHQKSNRQS6TR6Z6JL6HR47LYKLVQGY5WZR7Z4RBXC7NCLWYQ, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.303471950 (22h42m37.774682689s) [noise=true , address=145.79.227.34:4906 , piece_id=XNS7EVNDDK3XQVIYL5RPPIJPPTFNSTVJFWF6Q55EL3SRRAQTRYEQ, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.304017129 (22h42m37.775227868s) [noise=true , address=145.79.227.8:4330 , piece_id=3PCXR266AYJQSCQDCPBESLN7H4WLS6BLXDQWMWUOBS37SRVVKDCQ, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.304628322 (22h42m37.775839081s) [noise=true , address=77.90.31.11:4555 , piece_id=NDNU7HIDBMQGCP5HKQEGWAQCSENZQ4XPL5PDBLZSUVJNUMMKDKLA, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.305202977 (22h42m37.776413716s) [noise=true , address=145.79.227.27:4691 , piece_id=CIIRLVK6IODAMCRN7LBRLXQ5IR2AC7J3QAPHZUA3GROJIWNNEELQ, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.305742744 (22h42m37.776953483s) [noise=true , address=145.79.227.14:4840 , piece_id=7CFAXRK7F366A7QPI5OPTWMRO3Z5LK7LMEDP37AB5XLUMP34JVHQ, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.306643649 (22h42m37.777854389s) [noise=true , address=145.79.227.9:4367 , piece_id=E7UKP2XNQTELMOXMLN5CUTEG6ZVZE5TT4DW6ZYUD5733HRHWYD6Q, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.306719754 (22h42m37.777930493s) [noise=true , address=145.79.227.26:4655 , piece_id=O65JQUKSUPZSBOI5EPDDRSPMCN2LI2BYWBVDYSRIG4H6F2ASKS2A, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.307495471 (22h42m37.7787062s) [noise=true , address=145.79.227.7:4293 , piece_id=HYDMGVL463LIAO4N7JRFMFZSZ566C7DHHEMMVCPHTDUTEQYESKFQ, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.307859154 (22h42m37.779069883s) [noise=true , address=145.79.227.13:4804 , piece_id=ALPYEUHA2CMGWO6NFA7UBQUIJZTGE4NF7QYLNVAZQKYTJKABT6DA, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.308629862 (22h42m37.779840601s) [noise=true , address=145.79.227.16:4911 , piece_id=2M7N5623QL2BC6IMB3RJVRUBLB2SNZ66SWZY45N6WBYLCC56IZMA, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.310368402 (22h42m37.781579141s) [noise=true , address=145.79.227.21:4474 , piece_id=EQILGZ4H4R254DE7UW43GYCETN77D5FIRK4SE4Y36PPPSOKBSX5A, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.315008367 (22h42m37.786219106s) [noise=true , address=145.79.227.32:4834 , piece_id=CIJJAFXIJYJ6MHGVXTZ6FLU36QWE3HNVOLWW4IMUH5ZCWTL6ETZQ, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.317256889 (22h42m37.788467628s) [noise=true , address=145.79.227.31:4799 , piece_id=FBKOSDEM2H5ETZM6OOGLZCDGTT5RQSLMRZ2CLX2GOZRHRCMR2RNA, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:03.018370042 (22h42m39.489580781s) [upload=19952]: Next returned nil segment Dec 4 15:38:03.018386903 (22h42m39.489597642s) [upload=19952]: Waiting for error group managing segments... Dec 4 15:38:03.360528210 (22h42m39.831738939s) [noise=true , address=77.90.31.11:4555 , piece_id=NDNU7HIDBMQGCP5HKQEGWAQCSENZQ4XPL5PDBLZSUVJNUMMKDKLA, seg_pos={0 0}, upload=19952]: Done uploading piece. err: <nil> Dec 4 15:38:03.476269451 (22h42m39.94748018s) [noise=true , address=145.79.227.16:4911 , piece_id=2M7N5623QL2BC6IMB3RJVRUBLB2SNZ66SWZY45N6WBYLCC56IZMA, seg_pos={0 0}, upload=19952]: Done uploading piece. err: <nil> Dec 4 15:38:03.477241432 (22h42m39.948452171s) [noise=true , address=145.79.227.9:4367 , piece_id=E7UKP2XNQTELMOXMLN5CUTEG6ZVZE5TT4DW6ZYUD5733HRHWYD6Q, seg_pos={0 0}, upload=19952]: Done uploading piece. err: <nil> Dec 4 15:38:03.483785764 (22h42m39.954996503s) [noise=true , address=145.79.227.14:4840 , piece_id=7CFAXRK7F366A7QPI5OPTWMRO3Z5LK7LMEDP37AB5XLUMP34JVHQ, seg_pos={0 0}, upload=19952]: Done uploading piece. err: <nil> Dec 4 15:38:03.484258473 (22h42m39.955469212s) [noise=true , address=145.79.227.27:4691 , piece_id=CIIRLVK6IODAMCRN7LBRLXQ5IR2AC7J3QAPHZUA3GROJIWNNEELQ, seg_pos={0 0}, upload=19952]: Done uploading piece. err: <nil> Dec 4 15:38:03.487612530 (22h42m39.958823269s) [noise=true , address=145.79.227.23:4547 , piece_id=AGTUJHQKSNRQS6TR6Z6JL6HR47LYKLVQGY5WZR7Z4RBXC7NCLWYQ, seg_pos={0 0}, upload=19952]: Done uploading piece. err: <nil> Dec 4 15:38:03.495282255 (22h42m39.966492984s) [noise=true , address=145.79.227.40:4051 , piece_id=RLDHK2A262X6IZQWP6DK2IPWBTMXYZ4YQQ7SG4IJRSO3KDPBUJYA, seg_pos={0 0}, upload=19952]: Done uploading piece. err: <nil> Dec 4 15:38:03.509793507 (22h42m39.981004236s) [noise=true , address=145.79.227.8:4330 , piece_id=3PCXR266AYJQSCQDCPBESLN7H4WLS6BLXDQWMWUOBS37SRVVKDCQ, seg_pos={0 0}, upload=19952]: Done uploading piece. err: <nil> Dec 4 15:38:03.513640021 (22h42m39.98485076s) [noise=true , address=145.79.227.34:4906 , piece_id=XNS7EVNDDK3XQVIYL5RPPIJPPTFNSTVJFWF6Q55EL3SRRAQTRYEQ, seg_pos={0 0}, upload=19952]: Done uploading piece. err: <nil> Dec 4 15:38:03.785547720 (22h42m40.256758459s) [noise=true , address=145.79.227.13:4804 , piece_id=ALPYEUHA2CMGWO6NFA7UBQUIJZTGE4NF7QYLNVAZQKYTJKABT6DA, seg_pos={0 0}, upload=19952]: Done uploading piece. err: <nil> Dec 4 15:38:03.809124804 (22h42m40.280335543s) [noise=true , address=145.79.227.21:4474 , piece_id=EQILGZ4H4R254DE7UW43GYCETN77D5FIRK4SE4Y36PPPSOKBSX5A, seg_pos={0 0}, upload=19952]: Done uploading piece. err: <nil> Dec 4 15:38:04.138848766 (22h42m40.610059495s) [noise=true , address=145.79.227.7:4293 , piece_id=HYDMGVL463LIAO4N7JRFMFZSZ566C7DHHEMMVCPHTDUTEQYESKFQ, seg_pos={0 0}, upload=19952]: Done uploading piece. err: <nil> Dec 4 15:38:04.176120728 (22h42m40.647331457s) [noise=true , address=145.79.227.32:4834 , piece_id=CIJJAFXIJYJ6MHGVXTZ6FLU36QWE3HNVOLWW4IMUH5ZCWTL6ETZQ, seg_pos={0 0}, upload=19952]: Done uploading piece. err: <nil> Dec 4 15:38:04.176143252 (22h42m40.647353991s) [seg_pos={0 0}, upload=19952]: Segment reached optimal threshold of 13 pieces. Dec 4 15:38:04.176373089 (22h42m40.647583818s) [noise=true , address=145.79.227.26:4655 , piece_id=O65JQUKSUPZSBOI5EPDDRSPMCN2LI2BYWBVDYSRIG4H6F2ASKS2A, seg_pos={0 0}, upload=19952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xpZx5VbgHmbepaS9ZXgHfp7SZGeYPEjuJddQeYPUVpDU4EGx5): protocol: write tcp 188.166.225.170:46794->145.79.227.26:4655: use of closed network connection; write tcp 188.166.225.170:46794->145.79.227.26:4655: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46794->145.79.227.26:4655: use of closed network connection Dec 4 15:38:04.176382406 (22h42m40.647593145s) [noise=true , address=145.79.227.26:4655 , piece_id=O65JQUKSUPZSBOI5EPDDRSPMCN2LI2BYWBVDYSRIG4H6F2ASKS2A, seg_pos={0 0}, upload=19952]: Long tail cancellation due to optimal threshold Dec 4 15:38:04.176643925 (22h42m40.647854654s) [noise=true , address=145.79.227.31:4799 , piece_id=FBKOSDEM2H5ETZM6OOGLZCDGTT5RQSLMRZ2CLX2GOZRHRCMR2RNA, seg_pos={0 0}, upload=19952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X): protocol: expected piece hash; context canceled; EOF Dec 4 15:38:04.176651369 (22h42m40.647862108s) [noise=true , address=145.79.227.31:4799 , piece_id=FBKOSDEM2H5ETZM6OOGLZCDGTT5RQSLMRZ2CLX2GOZRHRCMR2RNA, seg_pos={0 0}, upload=19952]: Long tail cancellation due to optimal threshold Dec 4 15:38:04.176821292 (22h42m40.648032041s) [noise=true , address=145.79.227.42:4121 , piece_id=USB7H5RUYH4JMFQQRO5M3BSKB4NETHDTSGV5CJRFOT76D32V6DHQ, seg_pos={0 0}, upload=19952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh): protocol: expected piece hash; context canceled; EOF Dec 4 15:38:04.176869694 (22h42m40.648080433s) [noise=true , address=145.79.227.42:4121 , piece_id=USB7H5RUYH4JMFQQRO5M3BSKB4NETHDTSGV5CJRFOT76D32V6DHQ, seg_pos={0 0}, upload=19952]: Long tail cancellation due to optimal threshold Dec 4 15:38:04.176908829 (22h42m40.648119569s) [seg_pos={0 0}, upload=19952]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 4 15:38:04.177077430 (22h42m40.648288169s) [upload=19952]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 4 15:38:04.177087099 (22h42m40.648297848s) [upload=19952]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 4 15:38:04.502975285 (22h42m40.974186014s) [upload=19952]: Done uploading segments... Dec 4 15:38:04.502992749 (22h42m40.974203488s) [upload=19952]: Upload finished. err: <nil> Duration: 5.664273891s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 5.7321656s trace: https://jaeger.dev.storj.io/trace/1729cf59b626c72a FAIL (2025-12-04 16:16:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw, Piece ID: XNS7EVNDDK3XQVIYL5RPPIJPPTFNSTVJFWF6Q55EL3SRRAQTRYEQ) context deadline exceeded; piecestore: (Node ID: 12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, Piece ID: HYDMGVL463LIAO4N7JRFMFZSZ566C7DHHEMMVCPHTDUTEQYESKFQ) context deadline exceeded; piecestore: (Node ID: 1rL2DnjSgHP36Md3bCgy5padqTAhv3iQCeURbLMuuuMBxtUNZH, Piece ID: EQILGZ4H4R254DE7UW43GYCETN77D5FIRK4SE4Y36PPPSOKBSX5A) context deadline exceeded; piecestore: (Node ID: 12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n, Piece ID: RLDHK2A262X6IZQWP6DK2IPWBTMXYZ4YQQ7SG4IJRSO3KDPBUJYA) context deadline exceeded ======LOGS====== Dec 4 15:38:01.034643304 (22h42m37.505854043s) [upload=19952]: Starting upload Dec 4 15:38:01.034798069 (22h42m37.506008818s) [upload=19952]: Done starting upload Dec 4 15:38:01.034989493 (22h42m37.506200232s) [upload=19952]: Uploading segments... Dec 4 15:38:01.035032334 (22h42m37.506243073s) [upload=19952]: Got next segment. Inline: false Dec 4 15:38:01.035060508 (22h42m37.506271237s) [upload=19952]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 15:38:01.035064726 (22h42m37.506275455s) [upload=19952]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 15:38:01.301722830 (22h42m37.772933559s) [seg_pos={0 0}, upload=19952]: Begin upload segment... Dec 4 15:38:01.301843599 (22h42m37.773054338s) [seg_pos={0 0}, upload=19952]: Done begin upload segment. Dec 4 15:38:01.301883024 (22h42m37.773093753s) [noise=true , address=145.79.227.40:4051 , piece_id=RLDHK2A262X6IZQWP6DK2IPWBTMXYZ4YQQ7SG4IJRSO3KDPBUJYA, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.301945323 (22h42m37.773156092s) [noise=true , address=145.79.227.42:4121 , piece_id=USB7H5RUYH4JMFQQRO5M3BSKB4NETHDTSGV5CJRFOT76D32V6DHQ, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.302924858 (22h42m37.774135597s) [noise=true , address=145.79.227.23:4547 , piece_id=AGTUJHQKSNRQS6TR6Z6JL6HR47LYKLVQGY5WZR7Z4RBXC7NCLWYQ, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.303471950 (22h42m37.774682689s) [noise=true , address=145.79.227.34:4906 , piece_id=XNS7EVNDDK3XQVIYL5RPPIJPPTFNSTVJFWF6Q55EL3SRRAQTRYEQ, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.304017129 (22h42m37.775227868s) [noise=true , address=145.79.227.8:4330 , piece_id=3PCXR266AYJQSCQDCPBESLN7H4WLS6BLXDQWMWUOBS37SRVVKDCQ, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.304628322 (22h42m37.775839081s) [noise=true , address=77.90.31.11:4555 , piece_id=NDNU7HIDBMQGCP5HKQEGWAQCSENZQ4XPL5PDBLZSUVJNUMMKDKLA, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.305202977 (22h42m37.776413716s) [noise=true , address=145.79.227.27:4691 , piece_id=CIIRLVK6IODAMCRN7LBRLXQ5IR2AC7J3QAPHZUA3GROJIWNNEELQ, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.305742744 (22h42m37.776953483s) [noise=true , address=145.79.227.14:4840 , piece_id=7CFAXRK7F366A7QPI5OPTWMRO3Z5LK7LMEDP37AB5XLUMP34JVHQ, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.306643649 (22h42m37.777854389s) [noise=true , address=145.79.227.9:4367 , piece_id=E7UKP2XNQTELMOXMLN5CUTEG6ZVZE5TT4DW6ZYUD5733HRHWYD6Q, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.306719754 (22h42m37.777930493s) [noise=true , address=145.79.227.26:4655 , piece_id=O65JQUKSUPZSBOI5EPDDRSPMCN2LI2BYWBVDYSRIG4H6F2ASKS2A, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.307495471 (22h42m37.7787062s) [noise=true , address=145.79.227.7:4293 , piece_id=HYDMGVL463LIAO4N7JRFMFZSZ566C7DHHEMMVCPHTDUTEQYESKFQ, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.307859154 (22h42m37.779069883s) [noise=true , address=145.79.227.13:4804 , piece_id=ALPYEUHA2CMGWO6NFA7UBQUIJZTGE4NF7QYLNVAZQKYTJKABT6DA, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.308629862 (22h42m37.779840601s) [noise=true , address=145.79.227.16:4911 , piece_id=2M7N5623QL2BC6IMB3RJVRUBLB2SNZ66SWZY45N6WBYLCC56IZMA, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.310368402 (22h42m37.781579141s) [noise=true , address=145.79.227.21:4474 , piece_id=EQILGZ4H4R254DE7UW43GYCETN77D5FIRK4SE4Y36PPPSOKBSX5A, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.315008367 (22h42m37.786219106s) [noise=true , address=145.79.227.32:4834 , piece_id=CIJJAFXIJYJ6MHGVXTZ6FLU36QWE3HNVOLWW4IMUH5ZCWTL6ETZQ, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.317256889 (22h42m37.788467628s) [noise=true , address=145.79.227.31:4799 , piece_id=FBKOSDEM2H5ETZM6OOGLZCDGTT5RQSLMRZ2CLX2GOZRHRCMR2RNA, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:03.018370042 (22h42m39.489580781s) [upload=19952]: Next returned nil segment Dec 4 15:38:03.018386903 (22h42m39.489597642s) [upload=19952]: Waiting for error group managing segments... Dec 4 15:38:03.360528210 (22h42m39.831738939s) [noise=true , address=77.90.31.11:4555 , piece_id=NDNU7HIDBMQGCP5HKQEGWAQCSENZQ4XPL5PDBLZSUVJNUMMKDKLA, seg_pos={0 0}, upload=19952]: Done uploading piece. err: <nil> Dec 4 15:38:03.476269451 (22h42m39.94748018s) [noise=true , address=145.79.227.16:4911 , piece_id=2M7N5623QL2BC6IMB3RJVRUBLB2SNZ66SWZY45N6WBYLCC56IZMA, seg_pos={0 0}, upload=19952]: Done uploading piece. err: <nil> Dec 4 15:38:03.477241432 (22h42m39.948452171s) [noise=true , address=145.79.227.9:4367 , piece_id=E7UKP2XNQTELMOXMLN5CUTEG6ZVZE5TT4DW6ZYUD5733HRHWYD6Q, seg_pos={0 0}, upload=19952]: Done uploading piece. err: <nil> Dec 4 15:38:03.483785764 (22h42m39.954996503s) [noise=true , address=145.79.227.14:4840 , piece_id=7CFAXRK7F366A7QPI5OPTWMRO3Z5LK7LMEDP37AB5XLUMP34JVHQ, seg_pos={0 0}, upload=19952]: Done uploading piece. err: <nil> Dec 4 15:38:03.484258473 (22h42m39.955469212s) [noise=true , address=145.79.227.27:4691 , piece_id=CIIRLVK6IODAMCRN7LBRLXQ5IR2AC7J3QAPHZUA3GROJIWNNEELQ, seg_pos={0 0}, upload=19952]: Done uploading piece. err: <nil> Dec 4 15:38:03.487612530 (22h42m39.958823269s) [noise=true , address=145.79.227.23:4547 , piece_id=AGTUJHQKSNRQS6TR6Z6JL6HR47LYKLVQGY5WZR7Z4RBXC7NCLWYQ, seg_pos={0 0}, upload=19952]: Done uploading piece. err: <nil> Dec 4 15:38:03.495282255 (22h42m39.966492984s) [noise=true , address=145.79.227.40:4051 , piece_id=RLDHK2A262X6IZQWP6DK2IPWBTMXYZ4YQQ7SG4IJRSO3KDPBUJYA, seg_pos={0 0}, upload=19952]: Done uploading piece. err: <nil> Dec 4 15:38:03.509793507 (22h42m39.981004236s) [noise=true , address=145.79.227.8:4330 , piece_id=3PCXR266AYJQSCQDCPBESLN7H4WLS6BLXDQWMWUOBS37SRVVKDCQ, seg_pos={0 0}, upload=19952]: Done uploading piece. err: <nil> Dec 4 15:38:03.513640021 (22h42m39.98485076s) [noise=true , address=145.79.227.34:4906 , piece_id=XNS7EVNDDK3XQVIYL5RPPIJPPTFNSTVJFWF6Q55EL3SRRAQTRYEQ, seg_pos={0 0}, upload=19952]: Done uploading piece. err: <nil> Dec 4 15:38:03.785547720 (22h42m40.256758459s) [noise=true , address=145.79.227.13:4804 , piece_id=ALPYEUHA2CMGWO6NFA7UBQUIJZTGE4NF7QYLNVAZQKYTJKABT6DA, seg_pos={0 0}, upload=19952]: Done uploading piece. err: <nil> Dec 4 15:38:03.809124804 (22h42m40.280335543s) [noise=true , address=145.79.227.21:4474 , piece_id=EQILGZ4H4R254DE7UW43GYCETN77D5FIRK4SE4Y36PPPSOKBSX5A, seg_pos={0 0}, upload=19952]: Done uploading piece. err: <nil> Dec 4 15:38:04.138848766 (22h42m40.610059495s) [noise=true , address=145.79.227.7:4293 , piece_id=HYDMGVL463LIAO4N7JRFMFZSZ566C7DHHEMMVCPHTDUTEQYESKFQ, seg_pos={0 0}, upload=19952]: Done uploading piece. err: <nil> Dec 4 15:38:04.176120728 (22h42m40.647331457s) [noise=true , address=145.79.227.32:4834 , piece_id=CIJJAFXIJYJ6MHGVXTZ6FLU36QWE3HNVOLWW4IMUH5ZCWTL6ETZQ, seg_pos={0 0}, upload=19952]: Done uploading piece. err: <nil> Dec 4 15:38:04.176143252 (22h42m40.647353991s) [seg_pos={0 0}, upload=19952]: Segment reached optimal threshold of 13 pieces. Dec 4 15:38:04.176373089 (22h42m40.647583818s) [noise=true , address=145.79.227.26:4655 , piece_id=O65JQUKSUPZSBOI5EPDDRSPMCN2LI2BYWBVDYSRIG4H6F2ASKS2A, seg_pos={0 0}, upload=19952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xpZx5VbgHmbepaS9ZXgHfp7SZGeYPEjuJddQeYPUVpDU4EGx5): protocol: write tcp 188.166.225.170:46794->145.79.227.26:4655: use of closed network connection; write tcp 188.166.225.170:46794->145.79.227.26:4655: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46794->145.79.227.26:4655: use of closed network connection Dec 4 15:38:04.176382406 (22h42m40.647593145s) [noise=true , address=145.79.227.26:4655 , piece_id=O65JQUKSUPZSBOI5EPDDRSPMCN2LI2BYWBVDYSRIG4H6F2ASKS2A, seg_pos={0 0}, upload=19952]: Long tail cancellation due to optimal threshold Dec 4 15:38:04.176643925 (22h42m40.647854654s) [noise=true , address=145.79.227.31:4799 , piece_id=FBKOSDEM2H5ETZM6OOGLZCDGTT5RQSLMRZ2CLX2GOZRHRCMR2RNA, seg_pos={0 0}, upload=19952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X): protocol: expected piece hash; context canceled; EOF Dec 4 15:38:04.176651369 (22h42m40.647862108s) [noise=true , address=145.79.227.31:4799 , piece_id=FBKOSDEM2H5ETZM6OOGLZCDGTT5RQSLMRZ2CLX2GOZRHRCMR2RNA, seg_pos={0 0}, upload=19952]: Long tail cancellation due to optimal threshold Dec 4 15:38:04.176821292 (22h42m40.648032041s) [noise=true , address=145.79.227.42:4121 , piece_id=USB7H5RUYH4JMFQQRO5M3BSKB4NETHDTSGV5CJRFOT76D32V6DHQ, seg_pos={0 0}, upload=19952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh): protocol: expected piece hash; context canceled; EOF Dec 4 15:38:04.176869694 (22h42m40.648080433s) [noise=true , address=145.79.227.42:4121 , piece_id=USB7H5RUYH4JMFQQRO5M3BSKB4NETHDTSGV5CJRFOT76D32V6DHQ, seg_pos={0 0}, upload=19952]: Long tail cancellation due to optimal threshold Dec 4 15:38:04.176908829 (22h42m40.648119569s) [seg_pos={0 0}, upload=19952]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 4 15:38:04.177077430 (22h42m40.648288169s) [upload=19952]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 4 15:38:04.177087099 (22h42m40.648297848s) [upload=19952]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 4 15:38:04.502975285 (22h42m40.974186014s) [upload=19952]: Done uploading segments... Dec 4 15:38:04.502992749 (22h42m40.974203488s) [upload=19952]: Upload finished. err: <nil> Duration: 5.732186358s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 5.741965197s trace: https://jaeger.dev.storj.io/trace/1729cf59b626c72a FAIL (2025-12-04 16:15:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw, Piece ID: XNS7EVNDDK3XQVIYL5RPPIJPPTFNSTVJFWF6Q55EL3SRRAQTRYEQ) context deadline exceeded; piecestore: (Node ID: 12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, Piece ID: HYDMGVL463LIAO4N7JRFMFZSZ566C7DHHEMMVCPHTDUTEQYESKFQ) context deadline exceeded; piecestore: (Node ID: 1rL2DnjSgHP36Md3bCgy5padqTAhv3iQCeURbLMuuuMBxtUNZH, Piece ID: EQILGZ4H4R254DE7UW43GYCETN77D5FIRK4SE4Y36PPPSOKBSX5A) context deadline exceeded; piecestore: (Node ID: 12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n, Piece ID: RLDHK2A262X6IZQWP6DK2IPWBTMXYZ4YQQ7SG4IJRSO3KDPBUJYA) context deadline exceeded ======LOGS====== Dec 4 15:38:01.034643304 (22h42m37.505854043s) [upload=19952]: Starting upload Dec 4 15:38:01.034798069 (22h42m37.506008818s) [upload=19952]: Done starting upload Dec 4 15:38:01.034989493 (22h42m37.506200232s) [upload=19952]: Uploading segments... Dec 4 15:38:01.035032334 (22h42m37.506243073s) [upload=19952]: Got next segment. Inline: false Dec 4 15:38:01.035060508 (22h42m37.506271237s) [upload=19952]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 15:38:01.035064726 (22h42m37.506275455s) [upload=19952]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 15:38:01.301722830 (22h42m37.772933559s) [seg_pos={0 0}, upload=19952]: Begin upload segment... Dec 4 15:38:01.301843599 (22h42m37.773054338s) [seg_pos={0 0}, upload=19952]: Done begin upload segment. Dec 4 15:38:01.301883024 (22h42m37.773093753s) [noise=true , address=145.79.227.40:4051 , piece_id=RLDHK2A262X6IZQWP6DK2IPWBTMXYZ4YQQ7SG4IJRSO3KDPBUJYA, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.301945323 (22h42m37.773156092s) [noise=true , address=145.79.227.42:4121 , piece_id=USB7H5RUYH4JMFQQRO5M3BSKB4NETHDTSGV5CJRFOT76D32V6DHQ, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.302924858 (22h42m37.774135597s) [noise=true , address=145.79.227.23:4547 , piece_id=AGTUJHQKSNRQS6TR6Z6JL6HR47LYKLVQGY5WZR7Z4RBXC7NCLWYQ, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.303471950 (22h42m37.774682689s) [noise=true , address=145.79.227.34:4906 , piece_id=XNS7EVNDDK3XQVIYL5RPPIJPPTFNSTVJFWF6Q55EL3SRRAQTRYEQ, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.304017129 (22h42m37.775227868s) [noise=true , address=145.79.227.8:4330 , piece_id=3PCXR266AYJQSCQDCPBESLN7H4WLS6BLXDQWMWUOBS37SRVVKDCQ, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.304628322 (22h42m37.775839081s) [noise=true , address=77.90.31.11:4555 , piece_id=NDNU7HIDBMQGCP5HKQEGWAQCSENZQ4XPL5PDBLZSUVJNUMMKDKLA, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.305202977 (22h42m37.776413716s) [noise=true , address=145.79.227.27:4691 , piece_id=CIIRLVK6IODAMCRN7LBRLXQ5IR2AC7J3QAPHZUA3GROJIWNNEELQ, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.305742744 (22h42m37.776953483s) [noise=true , address=145.79.227.14:4840 , piece_id=7CFAXRK7F366A7QPI5OPTWMRO3Z5LK7LMEDP37AB5XLUMP34JVHQ, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.306643649 (22h42m37.777854389s) [noise=true , address=145.79.227.9:4367 , piece_id=E7UKP2XNQTELMOXMLN5CUTEG6ZVZE5TT4DW6ZYUD5733HRHWYD6Q, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.306719754 (22h42m37.777930493s) [noise=true , address=145.79.227.26:4655 , piece_id=O65JQUKSUPZSBOI5EPDDRSPMCN2LI2BYWBVDYSRIG4H6F2ASKS2A, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.307495471 (22h42m37.7787062s) [noise=true , address=145.79.227.7:4293 , piece_id=HYDMGVL463LIAO4N7JRFMFZSZ566C7DHHEMMVCPHTDUTEQYESKFQ, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.307859154 (22h42m37.779069883s) [noise=true , address=145.79.227.13:4804 , piece_id=ALPYEUHA2CMGWO6NFA7UBQUIJZTGE4NF7QYLNVAZQKYTJKABT6DA, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.308629862 (22h42m37.779840601s) [noise=true , address=145.79.227.16:4911 , piece_id=2M7N5623QL2BC6IMB3RJVRUBLB2SNZ66SWZY45N6WBYLCC56IZMA, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.310368402 (22h42m37.781579141s) [noise=true , address=145.79.227.21:4474 , piece_id=EQILGZ4H4R254DE7UW43GYCETN77D5FIRK4SE4Y36PPPSOKBSX5A, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.315008367 (22h42m37.786219106s) [noise=true , address=145.79.227.32:4834 , piece_id=CIJJAFXIJYJ6MHGVXTZ6FLU36QWE3HNVOLWW4IMUH5ZCWTL6ETZQ, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.317256889 (22h42m37.788467628s) [noise=true , address=145.79.227.31:4799 , piece_id=FBKOSDEM2H5ETZM6OOGLZCDGTT5RQSLMRZ2CLX2GOZRHRCMR2RNA, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:03.018370042 (22h42m39.489580781s) [upload=19952]: Next returned nil segment Dec 4 15:38:03.018386903 (22h42m39.489597642s) [upload=19952]: Waiting for error group managing segments... Dec 4 15:38:03.360528210 (22h42m39.831738939s) [noise=true , address=77.90.31.11:4555 , piece_id=NDNU7HIDBMQGCP5HKQEGWAQCSENZQ4XPL5PDBLZSUVJNUMMKDKLA, seg_pos={0 0}, upload=19952]: Done uploading piece. err: <nil> Dec 4 15:38:03.476269451 (22h42m39.94748018s) [noise=true , address=145.79.227.16:4911 , piece_id=2M7N5623QL2BC6IMB3RJVRUBLB2SNZ66SWZY45N6WBYLCC56IZMA, seg_pos={0 0}, upload=19952]: Done uploading piece. err: <nil> Dec 4 15:38:03.477241432 (22h42m39.948452171s) [noise=true , address=145.79.227.9:4367 , piece_id=E7UKP2XNQTELMOXMLN5CUTEG6ZVZE5TT4DW6ZYUD5733HRHWYD6Q, seg_pos={0 0}, upload=19952]: Done uploading piece. err: <nil> Dec 4 15:38:03.483785764 (22h42m39.954996503s) [noise=true , address=145.79.227.14:4840 , piece_id=7CFAXRK7F366A7QPI5OPTWMRO3Z5LK7LMEDP37AB5XLUMP34JVHQ, seg_pos={0 0}, upload=19952]: Done uploading piece. err: <nil> Dec 4 15:38:03.484258473 (22h42m39.955469212s) [noise=true , address=145.79.227.27:4691 , piece_id=CIIRLVK6IODAMCRN7LBRLXQ5IR2AC7J3QAPHZUA3GROJIWNNEELQ, seg_pos={0 0}, upload=19952]: Done uploading piece. err: <nil> Dec 4 15:38:03.487612530 (22h42m39.958823269s) [noise=true , address=145.79.227.23:4547 , piece_id=AGTUJHQKSNRQS6TR6Z6JL6HR47LYKLVQGY5WZR7Z4RBXC7NCLWYQ, seg_pos={0 0}, upload=19952]: Done uploading piece. err: <nil> Dec 4 15:38:03.495282255 (22h42m39.966492984s) [noise=true , address=145.79.227.40:4051 , piece_id=RLDHK2A262X6IZQWP6DK2IPWBTMXYZ4YQQ7SG4IJRSO3KDPBUJYA, seg_pos={0 0}, upload=19952]: Done uploading piece. err: <nil> Dec 4 15:38:03.509793507 (22h42m39.981004236s) [noise=true , address=145.79.227.8:4330 , piece_id=3PCXR266AYJQSCQDCPBESLN7H4WLS6BLXDQWMWUOBS37SRVVKDCQ, seg_pos={0 0}, upload=19952]: Done uploading piece. err: <nil> Dec 4 15:38:03.513640021 (22h42m39.98485076s) [noise=true , address=145.79.227.34:4906 , piece_id=XNS7EVNDDK3XQVIYL5RPPIJPPTFNSTVJFWF6Q55EL3SRRAQTRYEQ, seg_pos={0 0}, upload=19952]: Done uploading piece. err: <nil> Dec 4 15:38:03.785547720 (22h42m40.256758459s) [noise=true , address=145.79.227.13:4804 , piece_id=ALPYEUHA2CMGWO6NFA7UBQUIJZTGE4NF7QYLNVAZQKYTJKABT6DA, seg_pos={0 0}, upload=19952]: Done uploading piece. err: <nil> Dec 4 15:38:03.809124804 (22h42m40.280335543s) [noise=true , address=145.79.227.21:4474 , piece_id=EQILGZ4H4R254DE7UW43GYCETN77D5FIRK4SE4Y36PPPSOKBSX5A, seg_pos={0 0}, upload=19952]: Done uploading piece. err: <nil> Dec 4 15:38:04.138848766 (22h42m40.610059495s) [noise=true , address=145.79.227.7:4293 , piece_id=HYDMGVL463LIAO4N7JRFMFZSZ566C7DHHEMMVCPHTDUTEQYESKFQ, seg_pos={0 0}, upload=19952]: Done uploading piece. err: <nil> Dec 4 15:38:04.176120728 (22h42m40.647331457s) [noise=true , address=145.79.227.32:4834 , piece_id=CIJJAFXIJYJ6MHGVXTZ6FLU36QWE3HNVOLWW4IMUH5ZCWTL6ETZQ, seg_pos={0 0}, upload=19952]: Done uploading piece. err: <nil> Dec 4 15:38:04.176143252 (22h42m40.647353991s) [seg_pos={0 0}, upload=19952]: Segment reached optimal threshold of 13 pieces. Dec 4 15:38:04.176373089 (22h42m40.647583818s) [noise=true , address=145.79.227.26:4655 , piece_id=O65JQUKSUPZSBOI5EPDDRSPMCN2LI2BYWBVDYSRIG4H6F2ASKS2A, seg_pos={0 0}, upload=19952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xpZx5VbgHmbepaS9ZXgHfp7SZGeYPEjuJddQeYPUVpDU4EGx5): protocol: write tcp 188.166.225.170:46794->145.79.227.26:4655: use of closed network connection; write tcp 188.166.225.170:46794->145.79.227.26:4655: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46794->145.79.227.26:4655: use of closed network connection Dec 4 15:38:04.176382406 (22h42m40.647593145s) [noise=true , address=145.79.227.26:4655 , piece_id=O65JQUKSUPZSBOI5EPDDRSPMCN2LI2BYWBVDYSRIG4H6F2ASKS2A, seg_pos={0 0}, upload=19952]: Long tail cancellation due to optimal threshold Dec 4 15:38:04.176643925 (22h42m40.647854654s) [noise=true , address=145.79.227.31:4799 , piece_id=FBKOSDEM2H5ETZM6OOGLZCDGTT5RQSLMRZ2CLX2GOZRHRCMR2RNA, seg_pos={0 0}, upload=19952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X): protocol: expected piece hash; context canceled; EOF Dec 4 15:38:04.176651369 (22h42m40.647862108s) [noise=true , address=145.79.227.31:4799 , piece_id=FBKOSDEM2H5ETZM6OOGLZCDGTT5RQSLMRZ2CLX2GOZRHRCMR2RNA, seg_pos={0 0}, upload=19952]: Long tail cancellation due to optimal threshold Dec 4 15:38:04.176821292 (22h42m40.648032041s) [noise=true , address=145.79.227.42:4121 , piece_id=USB7H5RUYH4JMFQQRO5M3BSKB4NETHDTSGV5CJRFOT76D32V6DHQ, seg_pos={0 0}, upload=19952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh): protocol: expected piece hash; context canceled; EOF Dec 4 15:38:04.176869694 (22h42m40.648080433s) [noise=true , address=145.79.227.42:4121 , piece_id=USB7H5RUYH4JMFQQRO5M3BSKB4NETHDTSGV5CJRFOT76D32V6DHQ, seg_pos={0 0}, upload=19952]: Long tail cancellation due to optimal threshold Dec 4 15:38:04.176908829 (22h42m40.648119569s) [seg_pos={0 0}, upload=19952]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 4 15:38:04.177077430 (22h42m40.648288169s) [upload=19952]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 4 15:38:04.177087099 (22h42m40.648297848s) [upload=19952]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 4 15:38:04.502975285 (22h42m40.974186014s) [upload=19952]: Done uploading segments... Dec 4 15:38:04.502992749 (22h42m40.974203488s) [upload=19952]: Upload finished. err: <nil> Duration: 5.741993602s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 5.702993186s trace: https://jaeger.dev.storj.io/trace/1729cf59b626c72a FAIL (2025-12-04 16:15:13 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw, Piece ID: XNS7EVNDDK3XQVIYL5RPPIJPPTFNSTVJFWF6Q55EL3SRRAQTRYEQ) context deadline exceeded; piecestore: (Node ID: 12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, Piece ID: HYDMGVL463LIAO4N7JRFMFZSZ566C7DHHEMMVCPHTDUTEQYESKFQ) context deadline exceeded; piecestore: (Node ID: 1rL2DnjSgHP36Md3bCgy5padqTAhv3iQCeURbLMuuuMBxtUNZH, Piece ID: EQILGZ4H4R254DE7UW43GYCETN77D5FIRK4SE4Y36PPPSOKBSX5A) context deadline exceeded; piecestore: (Node ID: 12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n, Piece ID: RLDHK2A262X6IZQWP6DK2IPWBTMXYZ4YQQ7SG4IJRSO3KDPBUJYA) context deadline exceeded ======LOGS====== Dec 4 15:38:01.034643304 (22h42m37.505854043s) [upload=19952]: Starting upload Dec 4 15:38:01.034798069 (22h42m37.506008818s) [upload=19952]: Done starting upload Dec 4 15:38:01.034989493 (22h42m37.506200232s) [upload=19952]: Uploading segments... Dec 4 15:38:01.035032334 (22h42m37.506243073s) [upload=19952]: Got next segment. Inline: false Dec 4 15:38:01.035060508 (22h42m37.506271237s) [upload=19952]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 15:38:01.035064726 (22h42m37.506275455s) [upload=19952]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 15:38:01.301722830 (22h42m37.772933559s) [seg_pos={0 0}, upload=19952]: Begin upload segment... Dec 4 15:38:01.301843599 (22h42m37.773054338s) [seg_pos={0 0}, upload=19952]: Done begin upload segment. Dec 4 15:38:01.301883024 (22h42m37.773093753s) [noise=true , address=145.79.227.40:4051 , piece_id=RLDHK2A262X6IZQWP6DK2IPWBTMXYZ4YQQ7SG4IJRSO3KDPBUJYA, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.301945323 (22h42m37.773156092s) [noise=true , address=145.79.227.42:4121 , piece_id=USB7H5RUYH4JMFQQRO5M3BSKB4NETHDTSGV5CJRFOT76D32V6DHQ, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.302924858 (22h42m37.774135597s) [noise=true , address=145.79.227.23:4547 , piece_id=AGTUJHQKSNRQS6TR6Z6JL6HR47LYKLVQGY5WZR7Z4RBXC7NCLWYQ, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.303471950 (22h42m37.774682689s) [noise=true , address=145.79.227.34:4906 , piece_id=XNS7EVNDDK3XQVIYL5RPPIJPPTFNSTVJFWF6Q55EL3SRRAQTRYEQ, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.304017129 (22h42m37.775227868s) [noise=true , address=145.79.227.8:4330 , piece_id=3PCXR266AYJQSCQDCPBESLN7H4WLS6BLXDQWMWUOBS37SRVVKDCQ, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.304628322 (22h42m37.775839081s) [noise=true , address=77.90.31.11:4555 , piece_id=NDNU7HIDBMQGCP5HKQEGWAQCSENZQ4XPL5PDBLZSUVJNUMMKDKLA, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.305202977 (22h42m37.776413716s) [noise=true , address=145.79.227.27:4691 , piece_id=CIIRLVK6IODAMCRN7LBRLXQ5IR2AC7J3QAPHZUA3GROJIWNNEELQ, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.305742744 (22h42m37.776953483s) [noise=true , address=145.79.227.14:4840 , piece_id=7CFAXRK7F366A7QPI5OPTWMRO3Z5LK7LMEDP37AB5XLUMP34JVHQ, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.306643649 (22h42m37.777854389s) [noise=true , address=145.79.227.9:4367 , piece_id=E7UKP2XNQTELMOXMLN5CUTEG6ZVZE5TT4DW6ZYUD5733HRHWYD6Q, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.306719754 (22h42m37.777930493s) [noise=true , address=145.79.227.26:4655 , piece_id=O65JQUKSUPZSBOI5EPDDRSPMCN2LI2BYWBVDYSRIG4H6F2ASKS2A, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.307495471 (22h42m37.7787062s) [noise=true , address=145.79.227.7:4293 , piece_id=HYDMGVL463LIAO4N7JRFMFZSZ566C7DHHEMMVCPHTDUTEQYESKFQ, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.307859154 (22h42m37.779069883s) [noise=true , address=145.79.227.13:4804 , piece_id=ALPYEUHA2CMGWO6NFA7UBQUIJZTGE4NF7QYLNVAZQKYTJKABT6DA, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.308629862 (22h42m37.779840601s) [noise=true , address=145.79.227.16:4911 , piece_id=2M7N5623QL2BC6IMB3RJVRUBLB2SNZ66SWZY45N6WBYLCC56IZMA, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.310368402 (22h42m37.781579141s) [noise=true , address=145.79.227.21:4474 , piece_id=EQILGZ4H4R254DE7UW43GYCETN77D5FIRK4SE4Y36PPPSOKBSX5A, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.315008367 (22h42m37.786219106s) [noise=true , address=145.79.227.32:4834 , piece_id=CIJJAFXIJYJ6MHGVXTZ6FLU36QWE3HNVOLWW4IMUH5ZCWTL6ETZQ, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.317256889 (22h42m37.788467628s) [noise=true , address=145.79.227.31:4799 , piece_id=FBKOSDEM2H5ETZM6OOGLZCDGTT5RQSLMRZ2CLX2GOZRHRCMR2RNA, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:03.018370042 (22h42m39.489580781s) [upload=19952]: Next returned nil segment Dec 4 15:38:03.018386903 (22h42m39.489597642s) [upload=19952]: Waiting for error group managing segments... Dec 4 15:38:03.360528210 (22h42m39.831738939s) [noise=true , address=77.90.31.11:4555 , piece_id=NDNU7HIDBMQGCP5HKQEGWAQCSENZQ4XPL5PDBLZSUVJNUMMKDKLA, seg_pos={0 0}, upload=19952]: Done uploading piece. err: <nil> Dec 4 15:38:03.476269451 (22h42m39.94748018s) [noise=true , address=145.79.227.16:4911 , piece_id=2M7N5623QL2BC6IMB3RJVRUBLB2SNZ66SWZY45N6WBYLCC56IZMA, seg_pos={0 0}, upload=19952]: Done uploading piece. err: <nil> Dec 4 15:38:03.477241432 (22h42m39.948452171s) [noise=true , address=145.79.227.9:4367 , piece_id=E7UKP2XNQTELMOXMLN5CUTEG6ZVZE5TT4DW6ZYUD5733HRHWYD6Q, seg_pos={0 0}, upload=19952]: Done uploading piece. err: <nil> Dec 4 15:38:03.483785764 (22h42m39.954996503s) [noise=true , address=145.79.227.14:4840 , piece_id=7CFAXRK7F366A7QPI5OPTWMRO3Z5LK7LMEDP37AB5XLUMP34JVHQ, seg_pos={0 0}, upload=19952]: Done uploading piece. err: <nil> Dec 4 15:38:03.484258473 (22h42m39.955469212s) [noise=true , address=145.79.227.27:4691 , piece_id=CIIRLVK6IODAMCRN7LBRLXQ5IR2AC7J3QAPHZUA3GROJIWNNEELQ, seg_pos={0 0}, upload=19952]: Done uploading piece. err: <nil> Dec 4 15:38:03.487612530 (22h42m39.958823269s) [noise=true , address=145.79.227.23:4547 , piece_id=AGTUJHQKSNRQS6TR6Z6JL6HR47LYKLVQGY5WZR7Z4RBXC7NCLWYQ, seg_pos={0 0}, upload=19952]: Done uploading piece. err: <nil> Dec 4 15:38:03.495282255 (22h42m39.966492984s) [noise=true , address=145.79.227.40:4051 , piece_id=RLDHK2A262X6IZQWP6DK2IPWBTMXYZ4YQQ7SG4IJRSO3KDPBUJYA, seg_pos={0 0}, upload=19952]: Done uploading piece. err: <nil> Dec 4 15:38:03.509793507 (22h42m39.981004236s) [noise=true , address=145.79.227.8:4330 , piece_id=3PCXR266AYJQSCQDCPBESLN7H4WLS6BLXDQWMWUOBS37SRVVKDCQ, seg_pos={0 0}, upload=19952]: Done uploading piece. err: <nil> Dec 4 15:38:03.513640021 (22h42m39.98485076s) [noise=true , address=145.79.227.34:4906 , piece_id=XNS7EVNDDK3XQVIYL5RPPIJPPTFNSTVJFWF6Q55EL3SRRAQTRYEQ, seg_pos={0 0}, upload=19952]: Done uploading piece. err: <nil> Dec 4 15:38:03.785547720 (22h42m40.256758459s) [noise=true , address=145.79.227.13:4804 , piece_id=ALPYEUHA2CMGWO6NFA7UBQUIJZTGE4NF7QYLNVAZQKYTJKABT6DA, seg_pos={0 0}, upload=19952]: Done uploading piece. err: <nil> Dec 4 15:38:03.809124804 (22h42m40.280335543s) [noise=true , address=145.79.227.21:4474 , piece_id=EQILGZ4H4R254DE7UW43GYCETN77D5FIRK4SE4Y36PPPSOKBSX5A, seg_pos={0 0}, upload=19952]: Done uploading piece. err: <nil> Dec 4 15:38:04.138848766 (22h42m40.610059495s) [noise=true , address=145.79.227.7:4293 , piece_id=HYDMGVL463LIAO4N7JRFMFZSZ566C7DHHEMMVCPHTDUTEQYESKFQ, seg_pos={0 0}, upload=19952]: Done uploading piece. err: <nil> Dec 4 15:38:04.176120728 (22h42m40.647331457s) [noise=true , address=145.79.227.32:4834 , piece_id=CIJJAFXIJYJ6MHGVXTZ6FLU36QWE3HNVOLWW4IMUH5ZCWTL6ETZQ, seg_pos={0 0}, upload=19952]: Done uploading piece. err: <nil> Dec 4 15:38:04.176143252 (22h42m40.647353991s) [seg_pos={0 0}, upload=19952]: Segment reached optimal threshold of 13 pieces. Dec 4 15:38:04.176373089 (22h42m40.647583818s) [noise=true , address=145.79.227.26:4655 , piece_id=O65JQUKSUPZSBOI5EPDDRSPMCN2LI2BYWBVDYSRIG4H6F2ASKS2A, seg_pos={0 0}, upload=19952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xpZx5VbgHmbepaS9ZXgHfp7SZGeYPEjuJddQeYPUVpDU4EGx5): protocol: write tcp 188.166.225.170:46794->145.79.227.26:4655: use of closed network connection; write tcp 188.166.225.170:46794->145.79.227.26:4655: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46794->145.79.227.26:4655: use of closed network connection Dec 4 15:38:04.176382406 (22h42m40.647593145s) [noise=true , address=145.79.227.26:4655 , piece_id=O65JQUKSUPZSBOI5EPDDRSPMCN2LI2BYWBVDYSRIG4H6F2ASKS2A, seg_pos={0 0}, upload=19952]: Long tail cancellation due to optimal threshold Dec 4 15:38:04.176643925 (22h42m40.647854654s) [noise=true , address=145.79.227.31:4799 , piece_id=FBKOSDEM2H5ETZM6OOGLZCDGTT5RQSLMRZ2CLX2GOZRHRCMR2RNA, seg_pos={0 0}, upload=19952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X): protocol: expected piece hash; context canceled; EOF Dec 4 15:38:04.176651369 (22h42m40.647862108s) [noise=true , address=145.79.227.31:4799 , piece_id=FBKOSDEM2H5ETZM6OOGLZCDGTT5RQSLMRZ2CLX2GOZRHRCMR2RNA, seg_pos={0 0}, upload=19952]: Long tail cancellation due to optimal threshold Dec 4 15:38:04.176821292 (22h42m40.648032041s) [noise=true , address=145.79.227.42:4121 , piece_id=USB7H5RUYH4JMFQQRO5M3BSKB4NETHDTSGV5CJRFOT76D32V6DHQ, seg_pos={0 0}, upload=19952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh): protocol: expected piece hash; context canceled; EOF Dec 4 15:38:04.176869694 (22h42m40.648080433s) [noise=true , address=145.79.227.42:4121 , piece_id=USB7H5RUYH4JMFQQRO5M3BSKB4NETHDTSGV5CJRFOT76D32V6DHQ, seg_pos={0 0}, upload=19952]: Long tail cancellation due to optimal threshold Dec 4 15:38:04.176908829 (22h42m40.648119569s) [seg_pos={0 0}, upload=19952]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 4 15:38:04.177077430 (22h42m40.648288169s) [upload=19952]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 4 15:38:04.177087099 (22h42m40.648297848s) [upload=19952]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 4 15:38:04.502975285 (22h42m40.974186014s) [upload=19952]: Done uploading segments... Dec 4 15:38:04.502992749 (22h42m40.974203488s) [upload=19952]: Upload finished. err: <nil> Duration: 5.703032642s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 5.925340018s trace: https://jaeger.dev.storj.io/trace/1729cf59b626c72a FAIL (2025-12-04 16:14:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw, Piece ID: XNS7EVNDDK3XQVIYL5RPPIJPPTFNSTVJFWF6Q55EL3SRRAQTRYEQ) context deadline exceeded; piecestore: (Node ID: 12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, Piece ID: HYDMGVL463LIAO4N7JRFMFZSZ566C7DHHEMMVCPHTDUTEQYESKFQ) context deadline exceeded; piecestore: (Node ID: 1rL2DnjSgHP36Md3bCgy5padqTAhv3iQCeURbLMuuuMBxtUNZH, Piece ID: EQILGZ4H4R254DE7UW43GYCETN77D5FIRK4SE4Y36PPPSOKBSX5A) context deadline exceeded; piecestore: (Node ID: 12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n, Piece ID: RLDHK2A262X6IZQWP6DK2IPWBTMXYZ4YQQ7SG4IJRSO3KDPBUJYA) context deadline exceeded ======LOGS====== Dec 4 15:38:01.034643304 (22h42m37.505854043s) [upload=19952]: Starting upload Dec 4 15:38:01.034798069 (22h42m37.506008818s) [upload=19952]: Done starting upload Dec 4 15:38:01.034989493 (22h42m37.506200232s) [upload=19952]: Uploading segments... Dec 4 15:38:01.035032334 (22h42m37.506243073s) [upload=19952]: Got next segment. Inline: false Dec 4 15:38:01.035060508 (22h42m37.506271237s) [upload=19952]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 15:38:01.035064726 (22h42m37.506275455s) [upload=19952]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 15:38:01.301722830 (22h42m37.772933559s) [seg_pos={0 0}, upload=19952]: Begin upload segment... Dec 4 15:38:01.301843599 (22h42m37.773054338s) [seg_pos={0 0}, upload=19952]: Done begin upload segment. Dec 4 15:38:01.301883024 (22h42m37.773093753s) [noise=true , address=145.79.227.40:4051 , piece_id=RLDHK2A262X6IZQWP6DK2IPWBTMXYZ4YQQ7SG4IJRSO3KDPBUJYA, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.301945323 (22h42m37.773156092s) [noise=true , address=145.79.227.42:4121 , piece_id=USB7H5RUYH4JMFQQRO5M3BSKB4NETHDTSGV5CJRFOT76D32V6DHQ, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.302924858 (22h42m37.774135597s) [noise=true , address=145.79.227.23:4547 , piece_id=AGTUJHQKSNRQS6TR6Z6JL6HR47LYKLVQGY5WZR7Z4RBXC7NCLWYQ, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.303471950 (22h42m37.774682689s) [noise=true , address=145.79.227.34:4906 , piece_id=XNS7EVNDDK3XQVIYL5RPPIJPPTFNSTVJFWF6Q55EL3SRRAQTRYEQ, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.304017129 (22h42m37.775227868s) [noise=true , address=145.79.227.8:4330 , piece_id=3PCXR266AYJQSCQDCPBESLN7H4WLS6BLXDQWMWUOBS37SRVVKDCQ, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.304628322 (22h42m37.775839081s) [noise=true , address=77.90.31.11:4555 , piece_id=NDNU7HIDBMQGCP5HKQEGWAQCSENZQ4XPL5PDBLZSUVJNUMMKDKLA, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.305202977 (22h42m37.776413716s) [noise=true , address=145.79.227.27:4691 , piece_id=CIIRLVK6IODAMCRN7LBRLXQ5IR2AC7J3QAPHZUA3GROJIWNNEELQ, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.305742744 (22h42m37.776953483s) [noise=true , address=145.79.227.14:4840 , piece_id=7CFAXRK7F366A7QPI5OPTWMRO3Z5LK7LMEDP37AB5XLUMP34JVHQ, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.306643649 (22h42m37.777854389s) [noise=true , address=145.79.227.9:4367 , piece_id=E7UKP2XNQTELMOXMLN5CUTEG6ZVZE5TT4DW6ZYUD5733HRHWYD6Q, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.306719754 (22h42m37.777930493s) [noise=true , address=145.79.227.26:4655 , piece_id=O65JQUKSUPZSBOI5EPDDRSPMCN2LI2BYWBVDYSRIG4H6F2ASKS2A, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.307495471 (22h42m37.7787062s) [noise=true , address=145.79.227.7:4293 , piece_id=HYDMGVL463LIAO4N7JRFMFZSZ566C7DHHEMMVCPHTDUTEQYESKFQ, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.307859154 (22h42m37.779069883s) [noise=true , address=145.79.227.13:4804 , piece_id=ALPYEUHA2CMGWO6NFA7UBQUIJZTGE4NF7QYLNVAZQKYTJKABT6DA, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.308629862 (22h42m37.779840601s) [noise=true , address=145.79.227.16:4911 , piece_id=2M7N5623QL2BC6IMB3RJVRUBLB2SNZ66SWZY45N6WBYLCC56IZMA, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.310368402 (22h42m37.781579141s) [noise=true , address=145.79.227.21:4474 , piece_id=EQILGZ4H4R254DE7UW43GYCETN77D5FIRK4SE4Y36PPPSOKBSX5A, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.315008367 (22h42m37.786219106s) [noise=true , address=145.79.227.32:4834 , piece_id=CIJJAFXIJYJ6MHGVXTZ6FLU36QWE3HNVOLWW4IMUH5ZCWTL6ETZQ, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.317256889 (22h42m37.788467628s) [noise=true , address=145.79.227.31:4799 , piece_id=FBKOSDEM2H5ETZM6OOGLZCDGTT5RQSLMRZ2CLX2GOZRHRCMR2RNA, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:03.018370042 (22h42m39.489580781s) [upload=19952]: Next returned nil segment Dec 4 15:38:03.018386903 (22h42m39.489597642s) [upload=19952]: Waiting for error group managing segments... Dec 4 15:38:03.360528210 (22h42m39.831738939s) [noise=true , address=77.90.31.11:4555 , piece_id=NDNU7HIDBMQGCP5HKQEGWAQCSENZQ4XPL5PDBLZSUVJNUMMKDKLA, seg_pos={0 0}, upload=19952]: Done uploading piece. err: <nil> Dec 4 15:38:03.476269451 (22h42m39.94748018s) [noise=true , address=145.79.227.16:4911 , piece_id=2M7N5623QL2BC6IMB3RJVRUBLB2SNZ66SWZY45N6WBYLCC56IZMA, seg_pos={0 0}, upload=19952]: Done uploading piece. err: <nil> Dec 4 15:38:03.477241432 (22h42m39.948452171s) [noise=true , address=145.79.227.9:4367 , piece_id=E7UKP2XNQTELMOXMLN5CUTEG6ZVZE5TT4DW6ZYUD5733HRHWYD6Q, seg_pos={0 0}, upload=19952]: Done uploading piece. err: <nil> Dec 4 15:38:03.483785764 (22h42m39.954996503s) [noise=true , address=145.79.227.14:4840 , piece_id=7CFAXRK7F366A7QPI5OPTWMRO3Z5LK7LMEDP37AB5XLUMP34JVHQ, seg_pos={0 0}, upload=19952]: Done uploading piece. err: <nil> Dec 4 15:38:03.484258473 (22h42m39.955469212s) [noise=true , address=145.79.227.27:4691 , piece_id=CIIRLVK6IODAMCRN7LBRLXQ5IR2AC7J3QAPHZUA3GROJIWNNEELQ, seg_pos={0 0}, upload=19952]: Done uploading piece. err: <nil> Dec 4 15:38:03.487612530 (22h42m39.958823269s) [noise=true , address=145.79.227.23:4547 , piece_id=AGTUJHQKSNRQS6TR6Z6JL6HR47LYKLVQGY5WZR7Z4RBXC7NCLWYQ, seg_pos={0 0}, upload=19952]: Done uploading piece. err: <nil> Dec 4 15:38:03.495282255 (22h42m39.966492984s) [noise=true , address=145.79.227.40:4051 , piece_id=RLDHK2A262X6IZQWP6DK2IPWBTMXYZ4YQQ7SG4IJRSO3KDPBUJYA, seg_pos={0 0}, upload=19952]: Done uploading piece. err: <nil> Dec 4 15:38:03.509793507 (22h42m39.981004236s) [noise=true , address=145.79.227.8:4330 , piece_id=3PCXR266AYJQSCQDCPBESLN7H4WLS6BLXDQWMWUOBS37SRVVKDCQ, seg_pos={0 0}, upload=19952]: Done uploading piece. err: <nil> Dec 4 15:38:03.513640021 (22h42m39.98485076s) [noise=true , address=145.79.227.34:4906 , piece_id=XNS7EVNDDK3XQVIYL5RPPIJPPTFNSTVJFWF6Q55EL3SRRAQTRYEQ, seg_pos={0 0}, upload=19952]: Done uploading piece. err: <nil> Dec 4 15:38:03.785547720 (22h42m40.256758459s) [noise=true , address=145.79.227.13:4804 , piece_id=ALPYEUHA2CMGWO6NFA7UBQUIJZTGE4NF7QYLNVAZQKYTJKABT6DA, seg_pos={0 0}, upload=19952]: Done uploading piece. err: <nil> Dec 4 15:38:03.809124804 (22h42m40.280335543s) [noise=true , address=145.79.227.21:4474 , piece_id=EQILGZ4H4R254DE7UW43GYCETN77D5FIRK4SE4Y36PPPSOKBSX5A, seg_pos={0 0}, upload=19952]: Done uploading piece. err: <nil> Dec 4 15:38:04.138848766 (22h42m40.610059495s) [noise=true , address=145.79.227.7:4293 , piece_id=HYDMGVL463LIAO4N7JRFMFZSZ566C7DHHEMMVCPHTDUTEQYESKFQ, seg_pos={0 0}, upload=19952]: Done uploading piece. err: <nil> Dec 4 15:38:04.176120728 (22h42m40.647331457s) [noise=true , address=145.79.227.32:4834 , piece_id=CIJJAFXIJYJ6MHGVXTZ6FLU36QWE3HNVOLWW4IMUH5ZCWTL6ETZQ, seg_pos={0 0}, upload=19952]: Done uploading piece. err: <nil> Dec 4 15:38:04.176143252 (22h42m40.647353991s) [seg_pos={0 0}, upload=19952]: Segment reached optimal threshold of 13 pieces. Dec 4 15:38:04.176373089 (22h42m40.647583818s) [noise=true , address=145.79.227.26:4655 , piece_id=O65JQUKSUPZSBOI5EPDDRSPMCN2LI2BYWBVDYSRIG4H6F2ASKS2A, seg_pos={0 0}, upload=19952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xpZx5VbgHmbepaS9ZXgHfp7SZGeYPEjuJddQeYPUVpDU4EGx5): protocol: write tcp 188.166.225.170:46794->145.79.227.26:4655: use of closed network connection; write tcp 188.166.225.170:46794->145.79.227.26:4655: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46794->145.79.227.26:4655: use of closed network connection Dec 4 15:38:04.176382406 (22h42m40.647593145s) [noise=true , address=145.79.227.26:4655 , piece_id=O65JQUKSUPZSBOI5EPDDRSPMCN2LI2BYWBVDYSRIG4H6F2ASKS2A, seg_pos={0 0}, upload=19952]: Long tail cancellation due to optimal threshold Dec 4 15:38:04.176643925 (22h42m40.647854654s) [noise=true , address=145.79.227.31:4799 , piece_id=FBKOSDEM2H5ETZM6OOGLZCDGTT5RQSLMRZ2CLX2GOZRHRCMR2RNA, seg_pos={0 0}, upload=19952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X): protocol: expected piece hash; context canceled; EOF Dec 4 15:38:04.176651369 (22h42m40.647862108s) [noise=true , address=145.79.227.31:4799 , piece_id=FBKOSDEM2H5ETZM6OOGLZCDGTT5RQSLMRZ2CLX2GOZRHRCMR2RNA, seg_pos={0 0}, upload=19952]: Long tail cancellation due to optimal threshold Dec 4 15:38:04.176821292 (22h42m40.648032041s) [noise=true , address=145.79.227.42:4121 , piece_id=USB7H5RUYH4JMFQQRO5M3BSKB4NETHDTSGV5CJRFOT76D32V6DHQ, seg_pos={0 0}, upload=19952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh): protocol: expected piece hash; context canceled; EOF Dec 4 15:38:04.176869694 (22h42m40.648080433s) [noise=true , address=145.79.227.42:4121 , piece_id=USB7H5RUYH4JMFQQRO5M3BSKB4NETHDTSGV5CJRFOT76D32V6DHQ, seg_pos={0 0}, upload=19952]: Long tail cancellation due to optimal threshold Dec 4 15:38:04.176908829 (22h42m40.648119569s) [seg_pos={0 0}, upload=19952]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 4 15:38:04.177077430 (22h42m40.648288169s) [upload=19952]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 4 15:38:04.177087099 (22h42m40.648297848s) [upload=19952]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 4 15:38:04.502975285 (22h42m40.974186014s) [upload=19952]: Done uploading segments... Dec 4 15:38:04.502992749 (22h42m40.974203488s) [upload=19952]: Upload finished. err: <nil> Duration: 5.925367151s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 5.742703959s trace: https://jaeger.dev.storj.io/trace/1729cf59b626c72a FAIL (2025-12-04 16:14:14 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw, Piece ID: XNS7EVNDDK3XQVIYL5RPPIJPPTFNSTVJFWF6Q55EL3SRRAQTRYEQ) context deadline exceeded; piecestore: (Node ID: 12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, Piece ID: HYDMGVL463LIAO4N7JRFMFZSZ566C7DHHEMMVCPHTDUTEQYESKFQ) context deadline exceeded; piecestore: (Node ID: 1rL2DnjSgHP36Md3bCgy5padqTAhv3iQCeURbLMuuuMBxtUNZH, Piece ID: EQILGZ4H4R254DE7UW43GYCETN77D5FIRK4SE4Y36PPPSOKBSX5A) context deadline exceeded; piecestore: (Node ID: 12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n, Piece ID: RLDHK2A262X6IZQWP6DK2IPWBTMXYZ4YQQ7SG4IJRSO3KDPBUJYA) context deadline exceeded ======LOGS====== Dec 4 15:38:01.034643304 (22h42m37.505854043s) [upload=19952]: Starting upload Dec 4 15:38:01.034798069 (22h42m37.506008818s) [upload=19952]: Done starting upload Dec 4 15:38:01.034989493 (22h42m37.506200232s) [upload=19952]: Uploading segments... Dec 4 15:38:01.035032334 (22h42m37.506243073s) [upload=19952]: Got next segment. Inline: false Dec 4 15:38:01.035060508 (22h42m37.506271237s) [upload=19952]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 15:38:01.035064726 (22h42m37.506275455s) [upload=19952]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 15:38:01.301722830 (22h42m37.772933559s) [seg_pos={0 0}, upload=19952]: Begin upload segment... Dec 4 15:38:01.301843599 (22h42m37.773054338s) [seg_pos={0 0}, upload=19952]: Done begin upload segment. Dec 4 15:38:01.301883024 (22h42m37.773093753s) [noise=true , address=145.79.227.40:4051 , piece_id=RLDHK2A262X6IZQWP6DK2IPWBTMXYZ4YQQ7SG4IJRSO3KDPBUJYA, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.301945323 (22h42m37.773156092s) [noise=true , address=145.79.227.42:4121 , piece_id=USB7H5RUYH4JMFQQRO5M3BSKB4NETHDTSGV5CJRFOT76D32V6DHQ, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.302924858 (22h42m37.774135597s) [noise=true , address=145.79.227.23:4547 , piece_id=AGTUJHQKSNRQS6TR6Z6JL6HR47LYKLVQGY5WZR7Z4RBXC7NCLWYQ, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.303471950 (22h42m37.774682689s) [noise=true , address=145.79.227.34:4906 , piece_id=XNS7EVNDDK3XQVIYL5RPPIJPPTFNSTVJFWF6Q55EL3SRRAQTRYEQ, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.304017129 (22h42m37.775227868s) [noise=true , address=145.79.227.8:4330 , piece_id=3PCXR266AYJQSCQDCPBESLN7H4WLS6BLXDQWMWUOBS37SRVVKDCQ, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.304628322 (22h42m37.775839081s) [noise=true , address=77.90.31.11:4555 , piece_id=NDNU7HIDBMQGCP5HKQEGWAQCSENZQ4XPL5PDBLZSUVJNUMMKDKLA, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.305202977 (22h42m37.776413716s) [noise=true , address=145.79.227.27:4691 , piece_id=CIIRLVK6IODAMCRN7LBRLXQ5IR2AC7J3QAPHZUA3GROJIWNNEELQ, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.305742744 (22h42m37.776953483s) [noise=true , address=145.79.227.14:4840 , piece_id=7CFAXRK7F366A7QPI5OPTWMRO3Z5LK7LMEDP37AB5XLUMP34JVHQ, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.306643649 (22h42m37.777854389s) [noise=true , address=145.79.227.9:4367 , piece_id=E7UKP2XNQTELMOXMLN5CUTEG6ZVZE5TT4DW6ZYUD5733HRHWYD6Q, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.306719754 (22h42m37.777930493s) [noise=true , address=145.79.227.26:4655 , piece_id=O65JQUKSUPZSBOI5EPDDRSPMCN2LI2BYWBVDYSRIG4H6F2ASKS2A, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.307495471 (22h42m37.7787062s) [noise=true , address=145.79.227.7:4293 , piece_id=HYDMGVL463LIAO4N7JRFMFZSZ566C7DHHEMMVCPHTDUTEQYESKFQ, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.307859154 (22h42m37.779069883s) [noise=true , address=145.79.227.13:4804 , piece_id=ALPYEUHA2CMGWO6NFA7UBQUIJZTGE4NF7QYLNVAZQKYTJKABT6DA, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.308629862 (22h42m37.779840601s) [noise=true , address=145.79.227.16:4911 , piece_id=2M7N5623QL2BC6IMB3RJVRUBLB2SNZ66SWZY45N6WBYLCC56IZMA, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.310368402 (22h42m37.781579141s) [noise=true , address=145.79.227.21:4474 , piece_id=EQILGZ4H4R254DE7UW43GYCETN77D5FIRK4SE4Y36PPPSOKBSX5A, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.315008367 (22h42m37.786219106s) [noise=true , address=145.79.227.32:4834 , piece_id=CIJJAFXIJYJ6MHGVXTZ6FLU36QWE3HNVOLWW4IMUH5ZCWTL6ETZQ, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.317256889 (22h42m37.788467628s) [noise=true , address=145.79.227.31:4799 , piece_id=FBKOSDEM2H5ETZM6OOGLZCDGTT5RQSLMRZ2CLX2GOZRHRCMR2RNA, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:03.018370042 (22h42m39.489580781s) [upload=19952]: Next returned nil segment Dec 4 15:38:03.018386903 (22h42m39.489597642s) [upload=19952]: Waiting for error group managing segments... Dec 4 15:38:03.360528210 (22h42m39.831738939s) [noise=true , address=77.90.31.11:4555 , piece_id=NDNU7HIDBMQGCP5HKQEGWAQCSENZQ4XPL5PDBLZSUVJNUMMKDKLA, seg_pos={0 0}, upload=19952]: Done uploading piece. err: <nil> Dec 4 15:38:03.476269451 (22h42m39.94748018s) [noise=true , address=145.79.227.16:4911 , piece_id=2M7N5623QL2BC6IMB3RJVRUBLB2SNZ66SWZY45N6WBYLCC56IZMA, seg_pos={0 0}, upload=19952]: Done uploading piece. err: <nil> Dec 4 15:38:03.477241432 (22h42m39.948452171s) [noise=true , address=145.79.227.9:4367 , piece_id=E7UKP2XNQTELMOXMLN5CUTEG6ZVZE5TT4DW6ZYUD5733HRHWYD6Q, seg_pos={0 0}, upload=19952]: Done uploading piece. err: <nil> Dec 4 15:38:03.483785764 (22h42m39.954996503s) [noise=true , address=145.79.227.14:4840 , piece_id=7CFAXRK7F366A7QPI5OPTWMRO3Z5LK7LMEDP37AB5XLUMP34JVHQ, seg_pos={0 0}, upload=19952]: Done uploading piece. err: <nil> Dec 4 15:38:03.484258473 (22h42m39.955469212s) [noise=true , address=145.79.227.27:4691 , piece_id=CIIRLVK6IODAMCRN7LBRLXQ5IR2AC7J3QAPHZUA3GROJIWNNEELQ, seg_pos={0 0}, upload=19952]: Done uploading piece. err: <nil> Dec 4 15:38:03.487612530 (22h42m39.958823269s) [noise=true , address=145.79.227.23:4547 , piece_id=AGTUJHQKSNRQS6TR6Z6JL6HR47LYKLVQGY5WZR7Z4RBXC7NCLWYQ, seg_pos={0 0}, upload=19952]: Done uploading piece. err: <nil> Dec 4 15:38:03.495282255 (22h42m39.966492984s) [noise=true , address=145.79.227.40:4051 , piece_id=RLDHK2A262X6IZQWP6DK2IPWBTMXYZ4YQQ7SG4IJRSO3KDPBUJYA, seg_pos={0 0}, upload=19952]: Done uploading piece. err: <nil> Dec 4 15:38:03.509793507 (22h42m39.981004236s) [noise=true , address=145.79.227.8:4330 , piece_id=3PCXR266AYJQSCQDCPBESLN7H4WLS6BLXDQWMWUOBS37SRVVKDCQ, seg_pos={0 0}, upload=19952]: Done uploading piece. err: <nil> Dec 4 15:38:03.513640021 (22h42m39.98485076s) [noise=true , address=145.79.227.34:4906 , piece_id=XNS7EVNDDK3XQVIYL5RPPIJPPTFNSTVJFWF6Q55EL3SRRAQTRYEQ, seg_pos={0 0}, upload=19952]: Done uploading piece. err: <nil> Dec 4 15:38:03.785547720 (22h42m40.256758459s) [noise=true , address=145.79.227.13:4804 , piece_id=ALPYEUHA2CMGWO6NFA7UBQUIJZTGE4NF7QYLNVAZQKYTJKABT6DA, seg_pos={0 0}, upload=19952]: Done uploading piece. err: <nil> Dec 4 15:38:03.809124804 (22h42m40.280335543s) [noise=true , address=145.79.227.21:4474 , piece_id=EQILGZ4H4R254DE7UW43GYCETN77D5FIRK4SE4Y36PPPSOKBSX5A, seg_pos={0 0}, upload=19952]: Done uploading piece. err: <nil> Dec 4 15:38:04.138848766 (22h42m40.610059495s) [noise=true , address=145.79.227.7:4293 , piece_id=HYDMGVL463LIAO4N7JRFMFZSZ566C7DHHEMMVCPHTDUTEQYESKFQ, seg_pos={0 0}, upload=19952]: Done uploading piece. err: <nil> Dec 4 15:38:04.176120728 (22h42m40.647331457s) [noise=true , address=145.79.227.32:4834 , piece_id=CIJJAFXIJYJ6MHGVXTZ6FLU36QWE3HNVOLWW4IMUH5ZCWTL6ETZQ, seg_pos={0 0}, upload=19952]: Done uploading piece. err: <nil> Dec 4 15:38:04.176143252 (22h42m40.647353991s) [seg_pos={0 0}, upload=19952]: Segment reached optimal threshold of 13 pieces. Dec 4 15:38:04.176373089 (22h42m40.647583818s) [noise=true , address=145.79.227.26:4655 , piece_id=O65JQUKSUPZSBOI5EPDDRSPMCN2LI2BYWBVDYSRIG4H6F2ASKS2A, seg_pos={0 0}, upload=19952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xpZx5VbgHmbepaS9ZXgHfp7SZGeYPEjuJddQeYPUVpDU4EGx5): protocol: write tcp 188.166.225.170:46794->145.79.227.26:4655: use of closed network connection; write tcp 188.166.225.170:46794->145.79.227.26:4655: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46794->145.79.227.26:4655: use of closed network connection Dec 4 15:38:04.176382406 (22h42m40.647593145s) [noise=true , address=145.79.227.26:4655 , piece_id=O65JQUKSUPZSBOI5EPDDRSPMCN2LI2BYWBVDYSRIG4H6F2ASKS2A, seg_pos={0 0}, upload=19952]: Long tail cancellation due to optimal threshold Dec 4 15:38:04.176643925 (22h42m40.647854654s) [noise=true , address=145.79.227.31:4799 , piece_id=FBKOSDEM2H5ETZM6OOGLZCDGTT5RQSLMRZ2CLX2GOZRHRCMR2RNA, seg_pos={0 0}, upload=19952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X): protocol: expected piece hash; context canceled; EOF Dec 4 15:38:04.176651369 (22h42m40.647862108s) [noise=true , address=145.79.227.31:4799 , piece_id=FBKOSDEM2H5ETZM6OOGLZCDGTT5RQSLMRZ2CLX2GOZRHRCMR2RNA, seg_pos={0 0}, upload=19952]: Long tail cancellation due to optimal threshold Dec 4 15:38:04.176821292 (22h42m40.648032041s) [noise=true , address=145.79.227.42:4121 , piece_id=USB7H5RUYH4JMFQQRO5M3BSKB4NETHDTSGV5CJRFOT76D32V6DHQ, seg_pos={0 0}, upload=19952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh): protocol: expected piece hash; context canceled; EOF Dec 4 15:38:04.176869694 (22h42m40.648080433s) [noise=true , address=145.79.227.42:4121 , piece_id=USB7H5RUYH4JMFQQRO5M3BSKB4NETHDTSGV5CJRFOT76D32V6DHQ, seg_pos={0 0}, upload=19952]: Long tail cancellation due to optimal threshold Dec 4 15:38:04.176908829 (22h42m40.648119569s) [seg_pos={0 0}, upload=19952]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 4 15:38:04.177077430 (22h42m40.648288169s) [upload=19952]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 4 15:38:04.177087099 (22h42m40.648297848s) [upload=19952]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 4 15:38:04.502975285 (22h42m40.974186014s) [upload=19952]: Done uploading segments... Dec 4 15:38:04.502992749 (22h42m40.974203488s) [upload=19952]: Upload finished. err: <nil> Duration: 5.742747912s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 5.734953877s trace: https://jaeger.dev.storj.io/trace/1729cf59b626c72a FAIL (2025-12-04 16:13:47 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw, Piece ID: XNS7EVNDDK3XQVIYL5RPPIJPPTFNSTVJFWF6Q55EL3SRRAQTRYEQ) context deadline exceeded; piecestore: (Node ID: 12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, Piece ID: HYDMGVL463LIAO4N7JRFMFZSZ566C7DHHEMMVCPHTDUTEQYESKFQ) context deadline exceeded; piecestore: (Node ID: 1rL2DnjSgHP36Md3bCgy5padqTAhv3iQCeURbLMuuuMBxtUNZH, Piece ID: EQILGZ4H4R254DE7UW43GYCETN77D5FIRK4SE4Y36PPPSOKBSX5A) context deadline exceeded; piecestore: (Node ID: 12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n, Piece ID: RLDHK2A262X6IZQWP6DK2IPWBTMXYZ4YQQ7SG4IJRSO3KDPBUJYA) context deadline exceeded ======LOGS====== Dec 4 15:38:01.034643304 (22h42m37.505854043s) [upload=19952]: Starting upload Dec 4 15:38:01.034798069 (22h42m37.506008818s) [upload=19952]: Done starting upload Dec 4 15:38:01.034989493 (22h42m37.506200232s) [upload=19952]: Uploading segments... Dec 4 15:38:01.035032334 (22h42m37.506243073s) [upload=19952]: Got next segment. Inline: false Dec 4 15:38:01.035060508 (22h42m37.506271237s) [upload=19952]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 15:38:01.035064726 (22h42m37.506275455s) [upload=19952]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 15:38:01.301722830 (22h42m37.772933559s) [seg_pos={0 0}, upload=19952]: Begin upload segment... Dec 4 15:38:01.301843599 (22h42m37.773054338s) [seg_pos={0 0}, upload=19952]: Done begin upload segment. Dec 4 15:38:01.301883024 (22h42m37.773093753s) [noise=true , address=145.79.227.40:4051 , piece_id=RLDHK2A262X6IZQWP6DK2IPWBTMXYZ4YQQ7SG4IJRSO3KDPBUJYA, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.301945323 (22h42m37.773156092s) [noise=true , address=145.79.227.42:4121 , piece_id=USB7H5RUYH4JMFQQRO5M3BSKB4NETHDTSGV5CJRFOT76D32V6DHQ, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.302924858 (22h42m37.774135597s) [noise=true , address=145.79.227.23:4547 , piece_id=AGTUJHQKSNRQS6TR6Z6JL6HR47LYKLVQGY5WZR7Z4RBXC7NCLWYQ, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.303471950 (22h42m37.774682689s) [noise=true , address=145.79.227.34:4906 , piece_id=XNS7EVNDDK3XQVIYL5RPPIJPPTFNSTVJFWF6Q55EL3SRRAQTRYEQ, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.304017129 (22h42m37.775227868s) [noise=true , address=145.79.227.8:4330 , piece_id=3PCXR266AYJQSCQDCPBESLN7H4WLS6BLXDQWMWUOBS37SRVVKDCQ, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.304628322 (22h42m37.775839081s) [noise=true , address=77.90.31.11:4555 , piece_id=NDNU7HIDBMQGCP5HKQEGWAQCSENZQ4XPL5PDBLZSUVJNUMMKDKLA, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.305202977 (22h42m37.776413716s) [noise=true , address=145.79.227.27:4691 , piece_id=CIIRLVK6IODAMCRN7LBRLXQ5IR2AC7J3QAPHZUA3GROJIWNNEELQ, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.305742744 (22h42m37.776953483s) [noise=true , address=145.79.227.14:4840 , piece_id=7CFAXRK7F366A7QPI5OPTWMRO3Z5LK7LMEDP37AB5XLUMP34JVHQ, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.306643649 (22h42m37.777854389s) [noise=true , address=145.79.227.9:4367 , piece_id=E7UKP2XNQTELMOXMLN5CUTEG6ZVZE5TT4DW6ZYUD5733HRHWYD6Q, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.306719754 (22h42m37.777930493s) [noise=true , address=145.79.227.26:4655 , piece_id=O65JQUKSUPZSBOI5EPDDRSPMCN2LI2BYWBVDYSRIG4H6F2ASKS2A, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.307495471 (22h42m37.7787062s) [noise=true , address=145.79.227.7:4293 , piece_id=HYDMGVL463LIAO4N7JRFMFZSZ566C7DHHEMMVCPHTDUTEQYESKFQ, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.307859154 (22h42m37.779069883s) [noise=true , address=145.79.227.13:4804 , piece_id=ALPYEUHA2CMGWO6NFA7UBQUIJZTGE4NF7QYLNVAZQKYTJKABT6DA, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.308629862 (22h42m37.779840601s) [noise=true , address=145.79.227.16:4911 , piece_id=2M7N5623QL2BC6IMB3RJVRUBLB2SNZ66SWZY45N6WBYLCC56IZMA, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.310368402 (22h42m37.781579141s) [noise=true , address=145.79.227.21:4474 , piece_id=EQILGZ4H4R254DE7UW43GYCETN77D5FIRK4SE4Y36PPPSOKBSX5A, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.315008367 (22h42m37.786219106s) [noise=true , address=145.79.227.32:4834 , piece_id=CIJJAFXIJYJ6MHGVXTZ6FLU36QWE3HNVOLWW4IMUH5ZCWTL6ETZQ, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.317256889 (22h42m37.788467628s) [noise=true , address=145.79.227.31:4799 , piece_id=FBKOSDEM2H5ETZM6OOGLZCDGTT5RQSLMRZ2CLX2GOZRHRCMR2RNA, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:03.018370042 (22h42m39.489580781s) [upload=19952]: Next returned nil segment Dec 4 15:38:03.018386903 (22h42m39.489597642s) [upload=19952]: Waiting for error group managing segments... Dec 4 15:38:03.360528210 (22h42m39.831738939s) [noise=true , address=77.90.31.11:4555 , piece_id=NDNU7HIDBMQGCP5HKQEGWAQCSENZQ4XPL5PDBLZSUVJNUMMKDKLA, seg_pos={0 0}, upload=19952]: Done uploading piece. err: <nil> Dec 4 15:38:03.476269451 (22h42m39.94748018s) [noise=true , address=145.79.227.16:4911 , piece_id=2M7N5623QL2BC6IMB3RJVRUBLB2SNZ66SWZY45N6WBYLCC56IZMA, seg_pos={0 0}, upload=19952]: Done uploading piece. err: <nil> Dec 4 15:38:03.477241432 (22h42m39.948452171s) [noise=true , address=145.79.227.9:4367 , piece_id=E7UKP2XNQTELMOXMLN5CUTEG6ZVZE5TT4DW6ZYUD5733HRHWYD6Q, seg_pos={0 0}, upload=19952]: Done uploading piece. err: <nil> Dec 4 15:38:03.483785764 (22h42m39.954996503s) [noise=true , address=145.79.227.14:4840 , piece_id=7CFAXRK7F366A7QPI5OPTWMRO3Z5LK7LMEDP37AB5XLUMP34JVHQ, seg_pos={0 0}, upload=19952]: Done uploading piece. err: <nil> Dec 4 15:38:03.484258473 (22h42m39.955469212s) [noise=true , address=145.79.227.27:4691 , piece_id=CIIRLVK6IODAMCRN7LBRLXQ5IR2AC7J3QAPHZUA3GROJIWNNEELQ, seg_pos={0 0}, upload=19952]: Done uploading piece. err: <nil> Dec 4 15:38:03.487612530 (22h42m39.958823269s) [noise=true , address=145.79.227.23:4547 , piece_id=AGTUJHQKSNRQS6TR6Z6JL6HR47LYKLVQGY5WZR7Z4RBXC7NCLWYQ, seg_pos={0 0}, upload=19952]: Done uploading piece. err: <nil> Dec 4 15:38:03.495282255 (22h42m39.966492984s) [noise=true , address=145.79.227.40:4051 , piece_id=RLDHK2A262X6IZQWP6DK2IPWBTMXYZ4YQQ7SG4IJRSO3KDPBUJYA, seg_pos={0 0}, upload=19952]: Done uploading piece. err: <nil> Dec 4 15:38:03.509793507 (22h42m39.981004236s) [noise=true , address=145.79.227.8:4330 , piece_id=3PCXR266AYJQSCQDCPBESLN7H4WLS6BLXDQWMWUOBS37SRVVKDCQ, seg_pos={0 0}, upload=19952]: Done uploading piece. err: <nil> Dec 4 15:38:03.513640021 (22h42m39.98485076s) [noise=true , address=145.79.227.34:4906 , piece_id=XNS7EVNDDK3XQVIYL5RPPIJPPTFNSTVJFWF6Q55EL3SRRAQTRYEQ, seg_pos={0 0}, upload=19952]: Done uploading piece. err: <nil> Dec 4 15:38:03.785547720 (22h42m40.256758459s) [noise=true , address=145.79.227.13:4804 , piece_id=ALPYEUHA2CMGWO6NFA7UBQUIJZTGE4NF7QYLNVAZQKYTJKABT6DA, seg_pos={0 0}, upload=19952]: Done uploading piece. err: <nil> Dec 4 15:38:03.809124804 (22h42m40.280335543s) [noise=true , address=145.79.227.21:4474 , piece_id=EQILGZ4H4R254DE7UW43GYCETN77D5FIRK4SE4Y36PPPSOKBSX5A, seg_pos={0 0}, upload=19952]: Done uploading piece. err: <nil> Dec 4 15:38:04.138848766 (22h42m40.610059495s) [noise=true , address=145.79.227.7:4293 , piece_id=HYDMGVL463LIAO4N7JRFMFZSZ566C7DHHEMMVCPHTDUTEQYESKFQ, seg_pos={0 0}, upload=19952]: Done uploading piece. err: <nil> Dec 4 15:38:04.176120728 (22h42m40.647331457s) [noise=true , address=145.79.227.32:4834 , piece_id=CIJJAFXIJYJ6MHGVXTZ6FLU36QWE3HNVOLWW4IMUH5ZCWTL6ETZQ, seg_pos={0 0}, upload=19952]: Done uploading piece. err: <nil> Dec 4 15:38:04.176143252 (22h42m40.647353991s) [seg_pos={0 0}, upload=19952]: Segment reached optimal threshold of 13 pieces. Dec 4 15:38:04.176373089 (22h42m40.647583818s) [noise=true , address=145.79.227.26:4655 , piece_id=O65JQUKSUPZSBOI5EPDDRSPMCN2LI2BYWBVDYSRIG4H6F2ASKS2A, seg_pos={0 0}, upload=19952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xpZx5VbgHmbepaS9ZXgHfp7SZGeYPEjuJddQeYPUVpDU4EGx5): protocol: write tcp 188.166.225.170:46794->145.79.227.26:4655: use of closed network connection; write tcp 188.166.225.170:46794->145.79.227.26:4655: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46794->145.79.227.26:4655: use of closed network connection Dec 4 15:38:04.176382406 (22h42m40.647593145s) [noise=true , address=145.79.227.26:4655 , piece_id=O65JQUKSUPZSBOI5EPDDRSPMCN2LI2BYWBVDYSRIG4H6F2ASKS2A, seg_pos={0 0}, upload=19952]: Long tail cancellation due to optimal threshold Dec 4 15:38:04.176643925 (22h42m40.647854654s) [noise=true , address=145.79.227.31:4799 , piece_id=FBKOSDEM2H5ETZM6OOGLZCDGTT5RQSLMRZ2CLX2GOZRHRCMR2RNA, seg_pos={0 0}, upload=19952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X): protocol: expected piece hash; context canceled; EOF Dec 4 15:38:04.176651369 (22h42m40.647862108s) [noise=true , address=145.79.227.31:4799 , piece_id=FBKOSDEM2H5ETZM6OOGLZCDGTT5RQSLMRZ2CLX2GOZRHRCMR2RNA, seg_pos={0 0}, upload=19952]: Long tail cancellation due to optimal threshold Dec 4 15:38:04.176821292 (22h42m40.648032041s) [noise=true , address=145.79.227.42:4121 , piece_id=USB7H5RUYH4JMFQQRO5M3BSKB4NETHDTSGV5CJRFOT76D32V6DHQ, seg_pos={0 0}, upload=19952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh): protocol: expected piece hash; context canceled; EOF Dec 4 15:38:04.176869694 (22h42m40.648080433s) [noise=true , address=145.79.227.42:4121 , piece_id=USB7H5RUYH4JMFQQRO5M3BSKB4NETHDTSGV5CJRFOT76D32V6DHQ, seg_pos={0 0}, upload=19952]: Long tail cancellation due to optimal threshold Dec 4 15:38:04.176908829 (22h42m40.648119569s) [seg_pos={0 0}, upload=19952]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 4 15:38:04.177077430 (22h42m40.648288169s) [upload=19952]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 4 15:38:04.177087099 (22h42m40.648297848s) [upload=19952]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 4 15:38:04.502975285 (22h42m40.974186014s) [upload=19952]: Done uploading segments... Dec 4 15:38:04.502992749 (22h42m40.974203488s) [upload=19952]: Upload finished. err: <nil> Duration: 5.734986368s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 5.662506647s trace: https://jaeger.dev.storj.io/trace/1729cf59b626c72a FAIL (2025-12-04 16:12:47 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw, Piece ID: XNS7EVNDDK3XQVIYL5RPPIJPPTFNSTVJFWF6Q55EL3SRRAQTRYEQ) context deadline exceeded; piecestore: (Node ID: 12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, Piece ID: HYDMGVL463LIAO4N7JRFMFZSZ566C7DHHEMMVCPHTDUTEQYESKFQ) context deadline exceeded; piecestore: (Node ID: 1rL2DnjSgHP36Md3bCgy5padqTAhv3iQCeURbLMuuuMBxtUNZH, Piece ID: EQILGZ4H4R254DE7UW43GYCETN77D5FIRK4SE4Y36PPPSOKBSX5A) context deadline exceeded; piecestore: (Node ID: 12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n, Piece ID: RLDHK2A262X6IZQWP6DK2IPWBTMXYZ4YQQ7SG4IJRSO3KDPBUJYA) context deadline exceeded ======LOGS====== Dec 4 15:38:01.034643304 (22h42m37.505854043s) [upload=19952]: Starting upload Dec 4 15:38:01.034798069 (22h42m37.506008818s) [upload=19952]: Done starting upload Dec 4 15:38:01.034989493 (22h42m37.506200232s) [upload=19952]: Uploading segments... Dec 4 15:38:01.035032334 (22h42m37.506243073s) [upload=19952]: Got next segment. Inline: false Dec 4 15:38:01.035060508 (22h42m37.506271237s) [upload=19952]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 15:38:01.035064726 (22h42m37.506275455s) [upload=19952]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 15:38:01.301722830 (22h42m37.772933559s) [seg_pos={0 0}, upload=19952]: Begin upload segment... Dec 4 15:38:01.301843599 (22h42m37.773054338s) [seg_pos={0 0}, upload=19952]: Done begin upload segment. Dec 4 15:38:01.301883024 (22h42m37.773093753s) [noise=true , address=145.79.227.40:4051 , piece_id=RLDHK2A262X6IZQWP6DK2IPWBTMXYZ4YQQ7SG4IJRSO3KDPBUJYA, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.301945323 (22h42m37.773156092s) [noise=true , address=145.79.227.42:4121 , piece_id=USB7H5RUYH4JMFQQRO5M3BSKB4NETHDTSGV5CJRFOT76D32V6DHQ, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.302924858 (22h42m37.774135597s) [noise=true , address=145.79.227.23:4547 , piece_id=AGTUJHQKSNRQS6TR6Z6JL6HR47LYKLVQGY5WZR7Z4RBXC7NCLWYQ, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.303471950 (22h42m37.774682689s) [noise=true , address=145.79.227.34:4906 , piece_id=XNS7EVNDDK3XQVIYL5RPPIJPPTFNSTVJFWF6Q55EL3SRRAQTRYEQ, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.304017129 (22h42m37.775227868s) [noise=true , address=145.79.227.8:4330 , piece_id=3PCXR266AYJQSCQDCPBESLN7H4WLS6BLXDQWMWUOBS37SRVVKDCQ, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.304628322 (22h42m37.775839081s) [noise=true , address=77.90.31.11:4555 , piece_id=NDNU7HIDBMQGCP5HKQEGWAQCSENZQ4XPL5PDBLZSUVJNUMMKDKLA, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.305202977 (22h42m37.776413716s) [noise=true , address=145.79.227.27:4691 , piece_id=CIIRLVK6IODAMCRN7LBRLXQ5IR2AC7J3QAPHZUA3GROJIWNNEELQ, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.305742744 (22h42m37.776953483s) [noise=true , address=145.79.227.14:4840 , piece_id=7CFAXRK7F366A7QPI5OPTWMRO3Z5LK7LMEDP37AB5XLUMP34JVHQ, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.306643649 (22h42m37.777854389s) [noise=true , address=145.79.227.9:4367 , piece_id=E7UKP2XNQTELMOXMLN5CUTEG6ZVZE5TT4DW6ZYUD5733HRHWYD6Q, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.306719754 (22h42m37.777930493s) [noise=true , address=145.79.227.26:4655 , piece_id=O65JQUKSUPZSBOI5EPDDRSPMCN2LI2BYWBVDYSRIG4H6F2ASKS2A, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.307495471 (22h42m37.7787062s) [noise=true , address=145.79.227.7:4293 , piece_id=HYDMGVL463LIAO4N7JRFMFZSZ566C7DHHEMMVCPHTDUTEQYESKFQ, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.307859154 (22h42m37.779069883s) [noise=true , address=145.79.227.13:4804 , piece_id=ALPYEUHA2CMGWO6NFA7UBQUIJZTGE4NF7QYLNVAZQKYTJKABT6DA, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.308629862 (22h42m37.779840601s) [noise=true , address=145.79.227.16:4911 , piece_id=2M7N5623QL2BC6IMB3RJVRUBLB2SNZ66SWZY45N6WBYLCC56IZMA, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.310368402 (22h42m37.781579141s) [noise=true , address=145.79.227.21:4474 , piece_id=EQILGZ4H4R254DE7UW43GYCETN77D5FIRK4SE4Y36PPPSOKBSX5A, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.315008367 (22h42m37.786219106s) [noise=true , address=145.79.227.32:4834 , piece_id=CIJJAFXIJYJ6MHGVXTZ6FLU36QWE3HNVOLWW4IMUH5ZCWTL6ETZQ, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.317256889 (22h42m37.788467628s) [noise=true , address=145.79.227.31:4799 , piece_id=FBKOSDEM2H5ETZM6OOGLZCDGTT5RQSLMRZ2CLX2GOZRHRCMR2RNA, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:03.018370042 (22h42m39.489580781s) [upload=19952]: Next returned nil segment Dec 4 15:38:03.018386903 (22h42m39.489597642s) [upload=19952]: Waiting for error group managing segments... Dec 4 15:38:03.360528210 (22h42m39.831738939s) [noise=true , address=77.90.31.11:4555 , piece_id=NDNU7HIDBMQGCP5HKQEGWAQCSENZQ4XPL5PDBLZSUVJNUMMKDKLA, seg_pos={0 0}, upload=19952]: Done uploading piece. err: <nil> Dec 4 15:38:03.476269451 (22h42m39.94748018s) [noise=true , address=145.79.227.16:4911 , piece_id=2M7N5623QL2BC6IMB3RJVRUBLB2SNZ66SWZY45N6WBYLCC56IZMA, seg_pos={0 0}, upload=19952]: Done uploading piece. err: <nil> Dec 4 15:38:03.477241432 (22h42m39.948452171s) [noise=true , address=145.79.227.9:4367 , piece_id=E7UKP2XNQTELMOXMLN5CUTEG6ZVZE5TT4DW6ZYUD5733HRHWYD6Q, seg_pos={0 0}, upload=19952]: Done uploading piece. err: <nil> Dec 4 15:38:03.483785764 (22h42m39.954996503s) [noise=true , address=145.79.227.14:4840 , piece_id=7CFAXRK7F366A7QPI5OPTWMRO3Z5LK7LMEDP37AB5XLUMP34JVHQ, seg_pos={0 0}, upload=19952]: Done uploading piece. err: <nil> Dec 4 15:38:03.484258473 (22h42m39.955469212s) [noise=true , address=145.79.227.27:4691 , piece_id=CIIRLVK6IODAMCRN7LBRLXQ5IR2AC7J3QAPHZUA3GROJIWNNEELQ, seg_pos={0 0}, upload=19952]: Done uploading piece. err: <nil> Dec 4 15:38:03.487612530 (22h42m39.958823269s) [noise=true , address=145.79.227.23:4547 , piece_id=AGTUJHQKSNRQS6TR6Z6JL6HR47LYKLVQGY5WZR7Z4RBXC7NCLWYQ, seg_pos={0 0}, upload=19952]: Done uploading piece. err: <nil> Dec 4 15:38:03.495282255 (22h42m39.966492984s) [noise=true , address=145.79.227.40:4051 , piece_id=RLDHK2A262X6IZQWP6DK2IPWBTMXYZ4YQQ7SG4IJRSO3KDPBUJYA, seg_pos={0 0}, upload=19952]: Done uploading piece. err: <nil> Dec 4 15:38:03.509793507 (22h42m39.981004236s) [noise=true , address=145.79.227.8:4330 , piece_id=3PCXR266AYJQSCQDCPBESLN7H4WLS6BLXDQWMWUOBS37SRVVKDCQ, seg_pos={0 0}, upload=19952]: Done uploading piece. err: <nil> Dec 4 15:38:03.513640021 (22h42m39.98485076s) [noise=true , address=145.79.227.34:4906 , piece_id=XNS7EVNDDK3XQVIYL5RPPIJPPTFNSTVJFWF6Q55EL3SRRAQTRYEQ, seg_pos={0 0}, upload=19952]: Done uploading piece. err: <nil> Dec 4 15:38:03.785547720 (22h42m40.256758459s) [noise=true , address=145.79.227.13:4804 , piece_id=ALPYEUHA2CMGWO6NFA7UBQUIJZTGE4NF7QYLNVAZQKYTJKABT6DA, seg_pos={0 0}, upload=19952]: Done uploading piece. err: <nil> Dec 4 15:38:03.809124804 (22h42m40.280335543s) [noise=true , address=145.79.227.21:4474 , piece_id=EQILGZ4H4R254DE7UW43GYCETN77D5FIRK4SE4Y36PPPSOKBSX5A, seg_pos={0 0}, upload=19952]: Done uploading piece. err: <nil> Dec 4 15:38:04.138848766 (22h42m40.610059495s) [noise=true , address=145.79.227.7:4293 , piece_id=HYDMGVL463LIAO4N7JRFMFZSZ566C7DHHEMMVCPHTDUTEQYESKFQ, seg_pos={0 0}, upload=19952]: Done uploading piece. err: <nil> Dec 4 15:38:04.176120728 (22h42m40.647331457s) [noise=true , address=145.79.227.32:4834 , piece_id=CIJJAFXIJYJ6MHGVXTZ6FLU36QWE3HNVOLWW4IMUH5ZCWTL6ETZQ, seg_pos={0 0}, upload=19952]: Done uploading piece. err: <nil> Dec 4 15:38:04.176143252 (22h42m40.647353991s) [seg_pos={0 0}, upload=19952]: Segment reached optimal threshold of 13 pieces. Dec 4 15:38:04.176373089 (22h42m40.647583818s) [noise=true , address=145.79.227.26:4655 , piece_id=O65JQUKSUPZSBOI5EPDDRSPMCN2LI2BYWBVDYSRIG4H6F2ASKS2A, seg_pos={0 0}, upload=19952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xpZx5VbgHmbepaS9ZXgHfp7SZGeYPEjuJddQeYPUVpDU4EGx5): protocol: write tcp 188.166.225.170:46794->145.79.227.26:4655: use of closed network connection; write tcp 188.166.225.170:46794->145.79.227.26:4655: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46794->145.79.227.26:4655: use of closed network connection Dec 4 15:38:04.176382406 (22h42m40.647593145s) [noise=true , address=145.79.227.26:4655 , piece_id=O65JQUKSUPZSBOI5EPDDRSPMCN2LI2BYWBVDYSRIG4H6F2ASKS2A, seg_pos={0 0}, upload=19952]: Long tail cancellation due to optimal threshold Dec 4 15:38:04.176643925 (22h42m40.647854654s) [noise=true , address=145.79.227.31:4799 , piece_id=FBKOSDEM2H5ETZM6OOGLZCDGTT5RQSLMRZ2CLX2GOZRHRCMR2RNA, seg_pos={0 0}, upload=19952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X): protocol: expected piece hash; context canceled; EOF Dec 4 15:38:04.176651369 (22h42m40.647862108s) [noise=true , address=145.79.227.31:4799 , piece_id=FBKOSDEM2H5ETZM6OOGLZCDGTT5RQSLMRZ2CLX2GOZRHRCMR2RNA, seg_pos={0 0}, upload=19952]: Long tail cancellation due to optimal threshold Dec 4 15:38:04.176821292 (22h42m40.648032041s) [noise=true , address=145.79.227.42:4121 , piece_id=USB7H5RUYH4JMFQQRO5M3BSKB4NETHDTSGV5CJRFOT76D32V6DHQ, seg_pos={0 0}, upload=19952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh): protocol: expected piece hash; context canceled; EOF Dec 4 15:38:04.176869694 (22h42m40.648080433s) [noise=true , address=145.79.227.42:4121 , piece_id=USB7H5RUYH4JMFQQRO5M3BSKB4NETHDTSGV5CJRFOT76D32V6DHQ, seg_pos={0 0}, upload=19952]: Long tail cancellation due to optimal threshold Dec 4 15:38:04.176908829 (22h42m40.648119569s) [seg_pos={0 0}, upload=19952]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 4 15:38:04.177077430 (22h42m40.648288169s) [upload=19952]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 4 15:38:04.177087099 (22h42m40.648297848s) [upload=19952]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 4 15:38:04.502975285 (22h42m40.974186014s) [upload=19952]: Done uploading segments... Dec 4 15:38:04.502992749 (22h42m40.974203488s) [upload=19952]: Upload finished. err: <nil> Duration: 5.662533827s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 5.64025325s trace: https://jaeger.dev.storj.io/trace/1729cf59b626c72a FAIL (2025-12-04 16:11:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw, Piece ID: XNS7EVNDDK3XQVIYL5RPPIJPPTFNSTVJFWF6Q55EL3SRRAQTRYEQ) context deadline exceeded; piecestore: (Node ID: 12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, Piece ID: HYDMGVL463LIAO4N7JRFMFZSZ566C7DHHEMMVCPHTDUTEQYESKFQ) context deadline exceeded; piecestore: (Node ID: 1rL2DnjSgHP36Md3bCgy5padqTAhv3iQCeURbLMuuuMBxtUNZH, Piece ID: EQILGZ4H4R254DE7UW43GYCETN77D5FIRK4SE4Y36PPPSOKBSX5A) context deadline exceeded; piecestore: (Node ID: 12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n, Piece ID: RLDHK2A262X6IZQWP6DK2IPWBTMXYZ4YQQ7SG4IJRSO3KDPBUJYA) context deadline exceeded ======LOGS====== Dec 4 15:38:01.034643304 (22h42m37.505854043s) [upload=19952]: Starting upload Dec 4 15:38:01.034798069 (22h42m37.506008818s) [upload=19952]: Done starting upload Dec 4 15:38:01.034989493 (22h42m37.506200232s) [upload=19952]: Uploading segments... Dec 4 15:38:01.035032334 (22h42m37.506243073s) [upload=19952]: Got next segment. Inline: false Dec 4 15:38:01.035060508 (22h42m37.506271237s) [upload=19952]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 15:38:01.035064726 (22h42m37.506275455s) [upload=19952]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 15:38:01.301722830 (22h42m37.772933559s) [seg_pos={0 0}, upload=19952]: Begin upload segment... Dec 4 15:38:01.301843599 (22h42m37.773054338s) [seg_pos={0 0}, upload=19952]: Done begin upload segment. Dec 4 15:38:01.301883024 (22h42m37.773093753s) [noise=true , address=145.79.227.40:4051 , piece_id=RLDHK2A262X6IZQWP6DK2IPWBTMXYZ4YQQ7SG4IJRSO3KDPBUJYA, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.301945323 (22h42m37.773156092s) [noise=true , address=145.79.227.42:4121 , piece_id=USB7H5RUYH4JMFQQRO5M3BSKB4NETHDTSGV5CJRFOT76D32V6DHQ, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.302924858 (22h42m37.774135597s) [noise=true , address=145.79.227.23:4547 , piece_id=AGTUJHQKSNRQS6TR6Z6JL6HR47LYKLVQGY5WZR7Z4RBXC7NCLWYQ, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.303471950 (22h42m37.774682689s) [noise=true , address=145.79.227.34:4906 , piece_id=XNS7EVNDDK3XQVIYL5RPPIJPPTFNSTVJFWF6Q55EL3SRRAQTRYEQ, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.304017129 (22h42m37.775227868s) [noise=true , address=145.79.227.8:4330 , piece_id=3PCXR266AYJQSCQDCPBESLN7H4WLS6BLXDQWMWUOBS37SRVVKDCQ, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.304628322 (22h42m37.775839081s) [noise=true , address=77.90.31.11:4555 , piece_id=NDNU7HIDBMQGCP5HKQEGWAQCSENZQ4XPL5PDBLZSUVJNUMMKDKLA, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.305202977 (22h42m37.776413716s) [noise=true , address=145.79.227.27:4691 , piece_id=CIIRLVK6IODAMCRN7LBRLXQ5IR2AC7J3QAPHZUA3GROJIWNNEELQ, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.305742744 (22h42m37.776953483s) [noise=true , address=145.79.227.14:4840 , piece_id=7CFAXRK7F366A7QPI5OPTWMRO3Z5LK7LMEDP37AB5XLUMP34JVHQ, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.306643649 (22h42m37.777854389s) [noise=true , address=145.79.227.9:4367 , piece_id=E7UKP2XNQTELMOXMLN5CUTEG6ZVZE5TT4DW6ZYUD5733HRHWYD6Q, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.306719754 (22h42m37.777930493s) [noise=true , address=145.79.227.26:4655 , piece_id=O65JQUKSUPZSBOI5EPDDRSPMCN2LI2BYWBVDYSRIG4H6F2ASKS2A, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.307495471 (22h42m37.7787062s) [noise=true , address=145.79.227.7:4293 , piece_id=HYDMGVL463LIAO4N7JRFMFZSZ566C7DHHEMMVCPHTDUTEQYESKFQ, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.307859154 (22h42m37.779069883s) [noise=true , address=145.79.227.13:4804 , piece_id=ALPYEUHA2CMGWO6NFA7UBQUIJZTGE4NF7QYLNVAZQKYTJKABT6DA, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.308629862 (22h42m37.779840601s) [noise=true , address=145.79.227.16:4911 , piece_id=2M7N5623QL2BC6IMB3RJVRUBLB2SNZ66SWZY45N6WBYLCC56IZMA, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.310368402 (22h42m37.781579141s) [noise=true , address=145.79.227.21:4474 , piece_id=EQILGZ4H4R254DE7UW43GYCETN77D5FIRK4SE4Y36PPPSOKBSX5A, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.315008367 (22h42m37.786219106s) [noise=true , address=145.79.227.32:4834 , piece_id=CIJJAFXIJYJ6MHGVXTZ6FLU36QWE3HNVOLWW4IMUH5ZCWTL6ETZQ, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.317256889 (22h42m37.788467628s) [noise=true , address=145.79.227.31:4799 , piece_id=FBKOSDEM2H5ETZM6OOGLZCDGTT5RQSLMRZ2CLX2GOZRHRCMR2RNA, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:03.018370042 (22h42m39.489580781s) [upload=19952]: Next returned nil segment Dec 4 15:38:03.018386903 (22h42m39.489597642s) [upload=19952]: Waiting for error group managing segments... Dec 4 15:38:03.360528210 (22h42m39.831738939s) [noise=true , address=77.90.31.11:4555 , piece_id=NDNU7HIDBMQGCP5HKQEGWAQCSENZQ4XPL5PDBLZSUVJNUMMKDKLA, seg_pos={0 0}, upload=19952]: Done uploading piece. err: <nil> Dec 4 15:38:03.476269451 (22h42m39.94748018s) [noise=true , address=145.79.227.16:4911 , piece_id=2M7N5623QL2BC6IMB3RJVRUBLB2SNZ66SWZY45N6WBYLCC56IZMA, seg_pos={0 0}, upload=19952]: Done uploading piece. err: <nil> Dec 4 15:38:03.477241432 (22h42m39.948452171s) [noise=true , address=145.79.227.9:4367 , piece_id=E7UKP2XNQTELMOXMLN5CUTEG6ZVZE5TT4DW6ZYUD5733HRHWYD6Q, seg_pos={0 0}, upload=19952]: Done uploading piece. err: <nil> Dec 4 15:38:03.483785764 (22h42m39.954996503s) [noise=true , address=145.79.227.14:4840 , piece_id=7CFAXRK7F366A7QPI5OPTWMRO3Z5LK7LMEDP37AB5XLUMP34JVHQ, seg_pos={0 0}, upload=19952]: Done uploading piece. err: <nil> Dec 4 15:38:03.484258473 (22h42m39.955469212s) [noise=true , address=145.79.227.27:4691 , piece_id=CIIRLVK6IODAMCRN7LBRLXQ5IR2AC7J3QAPHZUA3GROJIWNNEELQ, seg_pos={0 0}, upload=19952]: Done uploading piece. err: <nil> Dec 4 15:38:03.487612530 (22h42m39.958823269s) [noise=true , address=145.79.227.23:4547 , piece_id=AGTUJHQKSNRQS6TR6Z6JL6HR47LYKLVQGY5WZR7Z4RBXC7NCLWYQ, seg_pos={0 0}, upload=19952]: Done uploading piece. err: <nil> Dec 4 15:38:03.495282255 (22h42m39.966492984s) [noise=true , address=145.79.227.40:4051 , piece_id=RLDHK2A262X6IZQWP6DK2IPWBTMXYZ4YQQ7SG4IJRSO3KDPBUJYA, seg_pos={0 0}, upload=19952]: Done uploading piece. err: <nil> Dec 4 15:38:03.509793507 (22h42m39.981004236s) [noise=true , address=145.79.227.8:4330 , piece_id=3PCXR266AYJQSCQDCPBESLN7H4WLS6BLXDQWMWUOBS37SRVVKDCQ, seg_pos={0 0}, upload=19952]: Done uploading piece. err: <nil> Dec 4 15:38:03.513640021 (22h42m39.98485076s) [noise=true , address=145.79.227.34:4906 , piece_id=XNS7EVNDDK3XQVIYL5RPPIJPPTFNSTVJFWF6Q55EL3SRRAQTRYEQ, seg_pos={0 0}, upload=19952]: Done uploading piece. err: <nil> Dec 4 15:38:03.785547720 (22h42m40.256758459s) [noise=true , address=145.79.227.13:4804 , piece_id=ALPYEUHA2CMGWO6NFA7UBQUIJZTGE4NF7QYLNVAZQKYTJKABT6DA, seg_pos={0 0}, upload=19952]: Done uploading piece. err: <nil> Dec 4 15:38:03.809124804 (22h42m40.280335543s) [noise=true , address=145.79.227.21:4474 , piece_id=EQILGZ4H4R254DE7UW43GYCETN77D5FIRK4SE4Y36PPPSOKBSX5A, seg_pos={0 0}, upload=19952]: Done uploading piece. err: <nil> Dec 4 15:38:04.138848766 (22h42m40.610059495s) [noise=true , address=145.79.227.7:4293 , piece_id=HYDMGVL463LIAO4N7JRFMFZSZ566C7DHHEMMVCPHTDUTEQYESKFQ, seg_pos={0 0}, upload=19952]: Done uploading piece. err: <nil> Dec 4 15:38:04.176120728 (22h42m40.647331457s) [noise=true , address=145.79.227.32:4834 , piece_id=CIJJAFXIJYJ6MHGVXTZ6FLU36QWE3HNVOLWW4IMUH5ZCWTL6ETZQ, seg_pos={0 0}, upload=19952]: Done uploading piece. err: <nil> Dec 4 15:38:04.176143252 (22h42m40.647353991s) [seg_pos={0 0}, upload=19952]: Segment reached optimal threshold of 13 pieces. Dec 4 15:38:04.176373089 (22h42m40.647583818s) [noise=true , address=145.79.227.26:4655 , piece_id=O65JQUKSUPZSBOI5EPDDRSPMCN2LI2BYWBVDYSRIG4H6F2ASKS2A, seg_pos={0 0}, upload=19952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xpZx5VbgHmbepaS9ZXgHfp7SZGeYPEjuJddQeYPUVpDU4EGx5): protocol: write tcp 188.166.225.170:46794->145.79.227.26:4655: use of closed network connection; write tcp 188.166.225.170:46794->145.79.227.26:4655: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46794->145.79.227.26:4655: use of closed network connection Dec 4 15:38:04.176382406 (22h42m40.647593145s) [noise=true , address=145.79.227.26:4655 , piece_id=O65JQUKSUPZSBOI5EPDDRSPMCN2LI2BYWBVDYSRIG4H6F2ASKS2A, seg_pos={0 0}, upload=19952]: Long tail cancellation due to optimal threshold Dec 4 15:38:04.176643925 (22h42m40.647854654s) [noise=true , address=145.79.227.31:4799 , piece_id=FBKOSDEM2H5ETZM6OOGLZCDGTT5RQSLMRZ2CLX2GOZRHRCMR2RNA, seg_pos={0 0}, upload=19952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X): protocol: expected piece hash; context canceled; EOF Dec 4 15:38:04.176651369 (22h42m40.647862108s) [noise=true , address=145.79.227.31:4799 , piece_id=FBKOSDEM2H5ETZM6OOGLZCDGTT5RQSLMRZ2CLX2GOZRHRCMR2RNA, seg_pos={0 0}, upload=19952]: Long tail cancellation due to optimal threshold Dec 4 15:38:04.176821292 (22h42m40.648032041s) [noise=true , address=145.79.227.42:4121 , piece_id=USB7H5RUYH4JMFQQRO5M3BSKB4NETHDTSGV5CJRFOT76D32V6DHQ, seg_pos={0 0}, upload=19952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh): protocol: expected piece hash; context canceled; EOF Dec 4 15:38:04.176869694 (22h42m40.648080433s) [noise=true , address=145.79.227.42:4121 , piece_id=USB7H5RUYH4JMFQQRO5M3BSKB4NETHDTSGV5CJRFOT76D32V6DHQ, seg_pos={0 0}, upload=19952]: Long tail cancellation due to optimal threshold Dec 4 15:38:04.176908829 (22h42m40.648119569s) [seg_pos={0 0}, upload=19952]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 4 15:38:04.177077430 (22h42m40.648288169s) [upload=19952]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 4 15:38:04.177087099 (22h42m40.648297848s) [upload=19952]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 4 15:38:04.502975285 (22h42m40.974186014s) [upload=19952]: Done uploading segments... Dec 4 15:38:04.502992749 (22h42m40.974203488s) [upload=19952]: Upload finished. err: <nil> Duration: 5.640285701s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 5.769126197s trace: https://jaeger.dev.storj.io/trace/1729cf59b626c72a FAIL (2025-12-04 16:11:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw, Piece ID: XNS7EVNDDK3XQVIYL5RPPIJPPTFNSTVJFWF6Q55EL3SRRAQTRYEQ) context deadline exceeded; piecestore: (Node ID: 12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, Piece ID: HYDMGVL463LIAO4N7JRFMFZSZ566C7DHHEMMVCPHTDUTEQYESKFQ) context deadline exceeded; piecestore: (Node ID: 1rL2DnjSgHP36Md3bCgy5padqTAhv3iQCeURbLMuuuMBxtUNZH, Piece ID: EQILGZ4H4R254DE7UW43GYCETN77D5FIRK4SE4Y36PPPSOKBSX5A) context deadline exceeded; piecestore: (Node ID: 12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n, Piece ID: RLDHK2A262X6IZQWP6DK2IPWBTMXYZ4YQQ7SG4IJRSO3KDPBUJYA) context deadline exceeded ======LOGS====== Dec 4 15:38:01.034643304 (22h42m37.505854043s) [upload=19952]: Starting upload Dec 4 15:38:01.034798069 (22h42m37.506008818s) [upload=19952]: Done starting upload Dec 4 15:38:01.034989493 (22h42m37.506200232s) [upload=19952]: Uploading segments... Dec 4 15:38:01.035032334 (22h42m37.506243073s) [upload=19952]: Got next segment. Inline: false Dec 4 15:38:01.035060508 (22h42m37.506271237s) [upload=19952]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 15:38:01.035064726 (22h42m37.506275455s) [upload=19952]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 15:38:01.301722830 (22h42m37.772933559s) [seg_pos={0 0}, upload=19952]: Begin upload segment... Dec 4 15:38:01.301843599 (22h42m37.773054338s) [seg_pos={0 0}, upload=19952]: Done begin upload segment. Dec 4 15:38:01.301883024 (22h42m37.773093753s) [noise=true , address=145.79.227.40:4051 , piece_id=RLDHK2A262X6IZQWP6DK2IPWBTMXYZ4YQQ7SG4IJRSO3KDPBUJYA, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.301945323 (22h42m37.773156092s) [noise=true , address=145.79.227.42:4121 , piece_id=USB7H5RUYH4JMFQQRO5M3BSKB4NETHDTSGV5CJRFOT76D32V6DHQ, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.302924858 (22h42m37.774135597s) [noise=true , address=145.79.227.23:4547 , piece_id=AGTUJHQKSNRQS6TR6Z6JL6HR47LYKLVQGY5WZR7Z4RBXC7NCLWYQ, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.303471950 (22h42m37.774682689s) [noise=true , address=145.79.227.34:4906 , piece_id=XNS7EVNDDK3XQVIYL5RPPIJPPTFNSTVJFWF6Q55EL3SRRAQTRYEQ, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.304017129 (22h42m37.775227868s) [noise=true , address=145.79.227.8:4330 , piece_id=3PCXR266AYJQSCQDCPBESLN7H4WLS6BLXDQWMWUOBS37SRVVKDCQ, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.304628322 (22h42m37.775839081s) [noise=true , address=77.90.31.11:4555 , piece_id=NDNU7HIDBMQGCP5HKQEGWAQCSENZQ4XPL5PDBLZSUVJNUMMKDKLA, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.305202977 (22h42m37.776413716s) [noise=true , address=145.79.227.27:4691 , piece_id=CIIRLVK6IODAMCRN7LBRLXQ5IR2AC7J3QAPHZUA3GROJIWNNEELQ, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.305742744 (22h42m37.776953483s) [noise=true , address=145.79.227.14:4840 , piece_id=7CFAXRK7F366A7QPI5OPTWMRO3Z5LK7LMEDP37AB5XLUMP34JVHQ, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.306643649 (22h42m37.777854389s) [noise=true , address=145.79.227.9:4367 , piece_id=E7UKP2XNQTELMOXMLN5CUTEG6ZVZE5TT4DW6ZYUD5733HRHWYD6Q, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.306719754 (22h42m37.777930493s) [noise=true , address=145.79.227.26:4655 , piece_id=O65JQUKSUPZSBOI5EPDDRSPMCN2LI2BYWBVDYSRIG4H6F2ASKS2A, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.307495471 (22h42m37.7787062s) [noise=true , address=145.79.227.7:4293 , piece_id=HYDMGVL463LIAO4N7JRFMFZSZ566C7DHHEMMVCPHTDUTEQYESKFQ, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.307859154 (22h42m37.779069883s) [noise=true , address=145.79.227.13:4804 , piece_id=ALPYEUHA2CMGWO6NFA7UBQUIJZTGE4NF7QYLNVAZQKYTJKABT6DA, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.308629862 (22h42m37.779840601s) [noise=true , address=145.79.227.16:4911 , piece_id=2M7N5623QL2BC6IMB3RJVRUBLB2SNZ66SWZY45N6WBYLCC56IZMA, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.310368402 (22h42m37.781579141s) [noise=true , address=145.79.227.21:4474 , piece_id=EQILGZ4H4R254DE7UW43GYCETN77D5FIRK4SE4Y36PPPSOKBSX5A, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.315008367 (22h42m37.786219106s) [noise=true , address=145.79.227.32:4834 , piece_id=CIJJAFXIJYJ6MHGVXTZ6FLU36QWE3HNVOLWW4IMUH5ZCWTL6ETZQ, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.317256889 (22h42m37.788467628s) [noise=true , address=145.79.227.31:4799 , piece_id=FBKOSDEM2H5ETZM6OOGLZCDGTT5RQSLMRZ2CLX2GOZRHRCMR2RNA, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:03.018370042 (22h42m39.489580781s) [upload=19952]: Next returned nil segment Dec 4 15:38:03.018386903 (22h42m39.489597642s) [upload=19952]: Waiting for error group managing segments... Dec 4 15:38:03.360528210 (22h42m39.831738939s) [noise=true , address=77.90.31.11:4555 , piece_id=NDNU7HIDBMQGCP5HKQEGWAQCSENZQ4XPL5PDBLZSUVJNUMMKDKLA, seg_pos={0 0}, upload=19952]: Done uploading piece. err: <nil> Dec 4 15:38:03.476269451 (22h42m39.94748018s) [noise=true , address=145.79.227.16:4911 , piece_id=2M7N5623QL2BC6IMB3RJVRUBLB2SNZ66SWZY45N6WBYLCC56IZMA, seg_pos={0 0}, upload=19952]: Done uploading piece. err: <nil> Dec 4 15:38:03.477241432 (22h42m39.948452171s) [noise=true , address=145.79.227.9:4367 , piece_id=E7UKP2XNQTELMOXMLN5CUTEG6ZVZE5TT4DW6ZYUD5733HRHWYD6Q, seg_pos={0 0}, upload=19952]: Done uploading piece. err: <nil> Dec 4 15:38:03.483785764 (22h42m39.954996503s) [noise=true , address=145.79.227.14:4840 , piece_id=7CFAXRK7F366A7QPI5OPTWMRO3Z5LK7LMEDP37AB5XLUMP34JVHQ, seg_pos={0 0}, upload=19952]: Done uploading piece. err: <nil> Dec 4 15:38:03.484258473 (22h42m39.955469212s) [noise=true , address=145.79.227.27:4691 , piece_id=CIIRLVK6IODAMCRN7LBRLXQ5IR2AC7J3QAPHZUA3GROJIWNNEELQ, seg_pos={0 0}, upload=19952]: Done uploading piece. err: <nil> Dec 4 15:38:03.487612530 (22h42m39.958823269s) [noise=true , address=145.79.227.23:4547 , piece_id=AGTUJHQKSNRQS6TR6Z6JL6HR47LYKLVQGY5WZR7Z4RBXC7NCLWYQ, seg_pos={0 0}, upload=19952]: Done uploading piece. err: <nil> Dec 4 15:38:03.495282255 (22h42m39.966492984s) [noise=true , address=145.79.227.40:4051 , piece_id=RLDHK2A262X6IZQWP6DK2IPWBTMXYZ4YQQ7SG4IJRSO3KDPBUJYA, seg_pos={0 0}, upload=19952]: Done uploading piece. err: <nil> Dec 4 15:38:03.509793507 (22h42m39.981004236s) [noise=true , address=145.79.227.8:4330 , piece_id=3PCXR266AYJQSCQDCPBESLN7H4WLS6BLXDQWMWUOBS37SRVVKDCQ, seg_pos={0 0}, upload=19952]: Done uploading piece. err: <nil> Dec 4 15:38:03.513640021 (22h42m39.98485076s) [noise=true , address=145.79.227.34:4906 , piece_id=XNS7EVNDDK3XQVIYL5RPPIJPPTFNSTVJFWF6Q55EL3SRRAQTRYEQ, seg_pos={0 0}, upload=19952]: Done uploading piece. err: <nil> Dec 4 15:38:03.785547720 (22h42m40.256758459s) [noise=true , address=145.79.227.13:4804 , piece_id=ALPYEUHA2CMGWO6NFA7UBQUIJZTGE4NF7QYLNVAZQKYTJKABT6DA, seg_pos={0 0}, upload=19952]: Done uploading piece. err: <nil> Dec 4 15:38:03.809124804 (22h42m40.280335543s) [noise=true , address=145.79.227.21:4474 , piece_id=EQILGZ4H4R254DE7UW43GYCETN77D5FIRK4SE4Y36PPPSOKBSX5A, seg_pos={0 0}, upload=19952]: Done uploading piece. err: <nil> Dec 4 15:38:04.138848766 (22h42m40.610059495s) [noise=true , address=145.79.227.7:4293 , piece_id=HYDMGVL463LIAO4N7JRFMFZSZ566C7DHHEMMVCPHTDUTEQYESKFQ, seg_pos={0 0}, upload=19952]: Done uploading piece. err: <nil> Dec 4 15:38:04.176120728 (22h42m40.647331457s) [noise=true , address=145.79.227.32:4834 , piece_id=CIJJAFXIJYJ6MHGVXTZ6FLU36QWE3HNVOLWW4IMUH5ZCWTL6ETZQ, seg_pos={0 0}, upload=19952]: Done uploading piece. err: <nil> Dec 4 15:38:04.176143252 (22h42m40.647353991s) [seg_pos={0 0}, upload=19952]: Segment reached optimal threshold of 13 pieces. Dec 4 15:38:04.176373089 (22h42m40.647583818s) [noise=true , address=145.79.227.26:4655 , piece_id=O65JQUKSUPZSBOI5EPDDRSPMCN2LI2BYWBVDYSRIG4H6F2ASKS2A, seg_pos={0 0}, upload=19952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xpZx5VbgHmbepaS9ZXgHfp7SZGeYPEjuJddQeYPUVpDU4EGx5): protocol: write tcp 188.166.225.170:46794->145.79.227.26:4655: use of closed network connection; write tcp 188.166.225.170:46794->145.79.227.26:4655: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46794->145.79.227.26:4655: use of closed network connection Dec 4 15:38:04.176382406 (22h42m40.647593145s) [noise=true , address=145.79.227.26:4655 , piece_id=O65JQUKSUPZSBOI5EPDDRSPMCN2LI2BYWBVDYSRIG4H6F2ASKS2A, seg_pos={0 0}, upload=19952]: Long tail cancellation due to optimal threshold Dec 4 15:38:04.176643925 (22h42m40.647854654s) [noise=true , address=145.79.227.31:4799 , piece_id=FBKOSDEM2H5ETZM6OOGLZCDGTT5RQSLMRZ2CLX2GOZRHRCMR2RNA, seg_pos={0 0}, upload=19952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X): protocol: expected piece hash; context canceled; EOF Dec 4 15:38:04.176651369 (22h42m40.647862108s) [noise=true , address=145.79.227.31:4799 , piece_id=FBKOSDEM2H5ETZM6OOGLZCDGTT5RQSLMRZ2CLX2GOZRHRCMR2RNA, seg_pos={0 0}, upload=19952]: Long tail cancellation due to optimal threshold Dec 4 15:38:04.176821292 (22h42m40.648032041s) [noise=true , address=145.79.227.42:4121 , piece_id=USB7H5RUYH4JMFQQRO5M3BSKB4NETHDTSGV5CJRFOT76D32V6DHQ, seg_pos={0 0}, upload=19952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh): protocol: expected piece hash; context canceled; EOF Dec 4 15:38:04.176869694 (22h42m40.648080433s) [noise=true , address=145.79.227.42:4121 , piece_id=USB7H5RUYH4JMFQQRO5M3BSKB4NETHDTSGV5CJRFOT76D32V6DHQ, seg_pos={0 0}, upload=19952]: Long tail cancellation due to optimal threshold Dec 4 15:38:04.176908829 (22h42m40.648119569s) [seg_pos={0 0}, upload=19952]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 4 15:38:04.177077430 (22h42m40.648288169s) [upload=19952]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 4 15:38:04.177087099 (22h42m40.648297848s) [upload=19952]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 4 15:38:04.502975285 (22h42m40.974186014s) [upload=19952]: Done uploading segments... Dec 4 15:38:04.502992749 (22h42m40.974203488s) [upload=19952]: Upload finished. err: <nil> Duration: 5.769155403s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 6.004199554s trace: https://jaeger.dev.storj.io/trace/1729cf59b626c72a FAIL (2025-12-04 16:10:25 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw, Piece ID: XNS7EVNDDK3XQVIYL5RPPIJPPTFNSTVJFWF6Q55EL3SRRAQTRYEQ) context deadline exceeded; piecestore: (Node ID: 12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, Piece ID: HYDMGVL463LIAO4N7JRFMFZSZ566C7DHHEMMVCPHTDUTEQYESKFQ) context deadline exceeded; piecestore: (Node ID: 1rL2DnjSgHP36Md3bCgy5padqTAhv3iQCeURbLMuuuMBxtUNZH, Piece ID: EQILGZ4H4R254DE7UW43GYCETN77D5FIRK4SE4Y36PPPSOKBSX5A) context deadline exceeded; piecestore: (Node ID: 12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n, Piece ID: RLDHK2A262X6IZQWP6DK2IPWBTMXYZ4YQQ7SG4IJRSO3KDPBUJYA) context deadline exceeded ======LOGS====== Dec 4 15:38:01.034643304 (22h42m37.505854043s) [upload=19952]: Starting upload Dec 4 15:38:01.034798069 (22h42m37.506008818s) [upload=19952]: Done starting upload Dec 4 15:38:01.034989493 (22h42m37.506200232s) [upload=19952]: Uploading segments... Dec 4 15:38:01.035032334 (22h42m37.506243073s) [upload=19952]: Got next segment. Inline: false Dec 4 15:38:01.035060508 (22h42m37.506271237s) [upload=19952]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 15:38:01.035064726 (22h42m37.506275455s) [upload=19952]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 15:38:01.301722830 (22h42m37.772933559s) [seg_pos={0 0}, upload=19952]: Begin upload segment... Dec 4 15:38:01.301843599 (22h42m37.773054338s) [seg_pos={0 0}, upload=19952]: Done begin upload segment. Dec 4 15:38:01.301883024 (22h42m37.773093753s) [noise=true , address=145.79.227.40:4051 , piece_id=RLDHK2A262X6IZQWP6DK2IPWBTMXYZ4YQQ7SG4IJRSO3KDPBUJYA, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.301945323 (22h42m37.773156092s) [noise=true , address=145.79.227.42:4121 , piece_id=USB7H5RUYH4JMFQQRO5M3BSKB4NETHDTSGV5CJRFOT76D32V6DHQ, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.302924858 (22h42m37.774135597s) [noise=true , address=145.79.227.23:4547 , piece_id=AGTUJHQKSNRQS6TR6Z6JL6HR47LYKLVQGY5WZR7Z4RBXC7NCLWYQ, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.303471950 (22h42m37.774682689s) [noise=true , address=145.79.227.34:4906 , piece_id=XNS7EVNDDK3XQVIYL5RPPIJPPTFNSTVJFWF6Q55EL3SRRAQTRYEQ, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.304017129 (22h42m37.775227868s) [noise=true , address=145.79.227.8:4330 , piece_id=3PCXR266AYJQSCQDCPBESLN7H4WLS6BLXDQWMWUOBS37SRVVKDCQ, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.304628322 (22h42m37.775839081s) [noise=true , address=77.90.31.11:4555 , piece_id=NDNU7HIDBMQGCP5HKQEGWAQCSENZQ4XPL5PDBLZSUVJNUMMKDKLA, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.305202977 (22h42m37.776413716s) [noise=true , address=145.79.227.27:4691 , piece_id=CIIRLVK6IODAMCRN7LBRLXQ5IR2AC7J3QAPHZUA3GROJIWNNEELQ, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.305742744 (22h42m37.776953483s) [noise=true , address=145.79.227.14:4840 , piece_id=7CFAXRK7F366A7QPI5OPTWMRO3Z5LK7LMEDP37AB5XLUMP34JVHQ, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.306643649 (22h42m37.777854389s) [noise=true , address=145.79.227.9:4367 , piece_id=E7UKP2XNQTELMOXMLN5CUTEG6ZVZE5TT4DW6ZYUD5733HRHWYD6Q, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.306719754 (22h42m37.777930493s) [noise=true , address=145.79.227.26:4655 , piece_id=O65JQUKSUPZSBOI5EPDDRSPMCN2LI2BYWBVDYSRIG4H6F2ASKS2A, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.307495471 (22h42m37.7787062s) [noise=true , address=145.79.227.7:4293 , piece_id=HYDMGVL463LIAO4N7JRFMFZSZ566C7DHHEMMVCPHTDUTEQYESKFQ, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.307859154 (22h42m37.779069883s) [noise=true , address=145.79.227.13:4804 , piece_id=ALPYEUHA2CMGWO6NFA7UBQUIJZTGE4NF7QYLNVAZQKYTJKABT6DA, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.308629862 (22h42m37.779840601s) [noise=true , address=145.79.227.16:4911 , piece_id=2M7N5623QL2BC6IMB3RJVRUBLB2SNZ66SWZY45N6WBYLCC56IZMA, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.310368402 (22h42m37.781579141s) [noise=true , address=145.79.227.21:4474 , piece_id=EQILGZ4H4R254DE7UW43GYCETN77D5FIRK4SE4Y36PPPSOKBSX5A, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.315008367 (22h42m37.786219106s) [noise=true , address=145.79.227.32:4834 , piece_id=CIJJAFXIJYJ6MHGVXTZ6FLU36QWE3HNVOLWW4IMUH5ZCWTL6ETZQ, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.317256889 (22h42m37.788467628s) [noise=true , address=145.79.227.31:4799 , piece_id=FBKOSDEM2H5ETZM6OOGLZCDGTT5RQSLMRZ2CLX2GOZRHRCMR2RNA, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:03.018370042 (22h42m39.489580781s) [upload=19952]: Next returned nil segment Dec 4 15:38:03.018386903 (22h42m39.489597642s) [upload=19952]: Waiting for error group managing segments... Dec 4 15:38:03.360528210 (22h42m39.831738939s) [noise=true , address=77.90.31.11:4555 , piece_id=NDNU7HIDBMQGCP5HKQEGWAQCSENZQ4XPL5PDBLZSUVJNUMMKDKLA, seg_pos={0 0}, upload=19952]: Done uploading piece. err: <nil> Dec 4 15:38:03.476269451 (22h42m39.94748018s) [noise=true , address=145.79.227.16:4911 , piece_id=2M7N5623QL2BC6IMB3RJVRUBLB2SNZ66SWZY45N6WBYLCC56IZMA, seg_pos={0 0}, upload=19952]: Done uploading piece. err: <nil> Dec 4 15:38:03.477241432 (22h42m39.948452171s) [noise=true , address=145.79.227.9:4367 , piece_id=E7UKP2XNQTELMOXMLN5CUTEG6ZVZE5TT4DW6ZYUD5733HRHWYD6Q, seg_pos={0 0}, upload=19952]: Done uploading piece. err: <nil> Dec 4 15:38:03.483785764 (22h42m39.954996503s) [noise=true , address=145.79.227.14:4840 , piece_id=7CFAXRK7F366A7QPI5OPTWMRO3Z5LK7LMEDP37AB5XLUMP34JVHQ, seg_pos={0 0}, upload=19952]: Done uploading piece. err: <nil> Dec 4 15:38:03.484258473 (22h42m39.955469212s) [noise=true , address=145.79.227.27:4691 , piece_id=CIIRLVK6IODAMCRN7LBRLXQ5IR2AC7J3QAPHZUA3GROJIWNNEELQ, seg_pos={0 0}, upload=19952]: Done uploading piece. err: <nil> Dec 4 15:38:03.487612530 (22h42m39.958823269s) [noise=true , address=145.79.227.23:4547 , piece_id=AGTUJHQKSNRQS6TR6Z6JL6HR47LYKLVQGY5WZR7Z4RBXC7NCLWYQ, seg_pos={0 0}, upload=19952]: Done uploading piece. err: <nil> Dec 4 15:38:03.495282255 (22h42m39.966492984s) [noise=true , address=145.79.227.40:4051 , piece_id=RLDHK2A262X6IZQWP6DK2IPWBTMXYZ4YQQ7SG4IJRSO3KDPBUJYA, seg_pos={0 0}, upload=19952]: Done uploading piece. err: <nil> Dec 4 15:38:03.509793507 (22h42m39.981004236s) [noise=true , address=145.79.227.8:4330 , piece_id=3PCXR266AYJQSCQDCPBESLN7H4WLS6BLXDQWMWUOBS37SRVVKDCQ, seg_pos={0 0}, upload=19952]: Done uploading piece. err: <nil> Dec 4 15:38:03.513640021 (22h42m39.98485076s) [noise=true , address=145.79.227.34:4906 , piece_id=XNS7EVNDDK3XQVIYL5RPPIJPPTFNSTVJFWF6Q55EL3SRRAQTRYEQ, seg_pos={0 0}, upload=19952]: Done uploading piece. err: <nil> Dec 4 15:38:03.785547720 (22h42m40.256758459s) [noise=true , address=145.79.227.13:4804 , piece_id=ALPYEUHA2CMGWO6NFA7UBQUIJZTGE4NF7QYLNVAZQKYTJKABT6DA, seg_pos={0 0}, upload=19952]: Done uploading piece. err: <nil> Dec 4 15:38:03.809124804 (22h42m40.280335543s) [noise=true , address=145.79.227.21:4474 , piece_id=EQILGZ4H4R254DE7UW43GYCETN77D5FIRK4SE4Y36PPPSOKBSX5A, seg_pos={0 0}, upload=19952]: Done uploading piece. err: <nil> Dec 4 15:38:04.138848766 (22h42m40.610059495s) [noise=true , address=145.79.227.7:4293 , piece_id=HYDMGVL463LIAO4N7JRFMFZSZ566C7DHHEMMVCPHTDUTEQYESKFQ, seg_pos={0 0}, upload=19952]: Done uploading piece. err: <nil> Dec 4 15:38:04.176120728 (22h42m40.647331457s) [noise=true , address=145.79.227.32:4834 , piece_id=CIJJAFXIJYJ6MHGVXTZ6FLU36QWE3HNVOLWW4IMUH5ZCWTL6ETZQ, seg_pos={0 0}, upload=19952]: Done uploading piece. err: <nil> Dec 4 15:38:04.176143252 (22h42m40.647353991s) [seg_pos={0 0}, upload=19952]: Segment reached optimal threshold of 13 pieces. Dec 4 15:38:04.176373089 (22h42m40.647583818s) [noise=true , address=145.79.227.26:4655 , piece_id=O65JQUKSUPZSBOI5EPDDRSPMCN2LI2BYWBVDYSRIG4H6F2ASKS2A, seg_pos={0 0}, upload=19952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xpZx5VbgHmbepaS9ZXgHfp7SZGeYPEjuJddQeYPUVpDU4EGx5): protocol: write tcp 188.166.225.170:46794->145.79.227.26:4655: use of closed network connection; write tcp 188.166.225.170:46794->145.79.227.26:4655: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46794->145.79.227.26:4655: use of closed network connection Dec 4 15:38:04.176382406 (22h42m40.647593145s) [noise=true , address=145.79.227.26:4655 , piece_id=O65JQUKSUPZSBOI5EPDDRSPMCN2LI2BYWBVDYSRIG4H6F2ASKS2A, seg_pos={0 0}, upload=19952]: Long tail cancellation due to optimal threshold Dec 4 15:38:04.176643925 (22h42m40.647854654s) [noise=true , address=145.79.227.31:4799 , piece_id=FBKOSDEM2H5ETZM6OOGLZCDGTT5RQSLMRZ2CLX2GOZRHRCMR2RNA, seg_pos={0 0}, upload=19952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X): protocol: expected piece hash; context canceled; EOF Dec 4 15:38:04.176651369 (22h42m40.647862108s) [noise=true , address=145.79.227.31:4799 , piece_id=FBKOSDEM2H5ETZM6OOGLZCDGTT5RQSLMRZ2CLX2GOZRHRCMR2RNA, seg_pos={0 0}, upload=19952]: Long tail cancellation due to optimal threshold Dec 4 15:38:04.176821292 (22h42m40.648032041s) [noise=true , address=145.79.227.42:4121 , piece_id=USB7H5RUYH4JMFQQRO5M3BSKB4NETHDTSGV5CJRFOT76D32V6DHQ, seg_pos={0 0}, upload=19952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh): protocol: expected piece hash; context canceled; EOF Dec 4 15:38:04.176869694 (22h42m40.648080433s) [noise=true , address=145.79.227.42:4121 , piece_id=USB7H5RUYH4JMFQQRO5M3BSKB4NETHDTSGV5CJRFOT76D32V6DHQ, seg_pos={0 0}, upload=19952]: Long tail cancellation due to optimal threshold Dec 4 15:38:04.176908829 (22h42m40.648119569s) [seg_pos={0 0}, upload=19952]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 4 15:38:04.177077430 (22h42m40.648288169s) [upload=19952]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 4 15:38:04.177087099 (22h42m40.648297848s) [upload=19952]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 4 15:38:04.502975285 (22h42m40.974186014s) [upload=19952]: Done uploading segments... Dec 4 15:38:04.502992749 (22h42m40.974203488s) [upload=19952]: Upload finished. err: <nil> Duration: 6.004230202s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 5.899075999s trace: https://jaeger.dev.storj.io/trace/1729cf59b626c72a FAIL (2025-12-04 16:10:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw, Piece ID: XNS7EVNDDK3XQVIYL5RPPIJPPTFNSTVJFWF6Q55EL3SRRAQTRYEQ) context deadline exceeded; piecestore: (Node ID: 12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, Piece ID: HYDMGVL463LIAO4N7JRFMFZSZ566C7DHHEMMVCPHTDUTEQYESKFQ) context deadline exceeded; piecestore: (Node ID: 1rL2DnjSgHP36Md3bCgy5padqTAhv3iQCeURbLMuuuMBxtUNZH, Piece ID: EQILGZ4H4R254DE7UW43GYCETN77D5FIRK4SE4Y36PPPSOKBSX5A) context deadline exceeded; piecestore: (Node ID: 12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n, Piece ID: RLDHK2A262X6IZQWP6DK2IPWBTMXYZ4YQQ7SG4IJRSO3KDPBUJYA) context deadline exceeded ======LOGS====== Dec 4 15:38:01.034643304 (22h42m37.505854043s) [upload=19952]: Starting upload Dec 4 15:38:01.034798069 (22h42m37.506008818s) [upload=19952]: Done starting upload Dec 4 15:38:01.034989493 (22h42m37.506200232s) [upload=19952]: Uploading segments... Dec 4 15:38:01.035032334 (22h42m37.506243073s) [upload=19952]: Got next segment. Inline: false Dec 4 15:38:01.035060508 (22h42m37.506271237s) [upload=19952]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 15:38:01.035064726 (22h42m37.506275455s) [upload=19952]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 15:38:01.301722830 (22h42m37.772933559s) [seg_pos={0 0}, upload=19952]: Begin upload segment... Dec 4 15:38:01.301843599 (22h42m37.773054338s) [seg_pos={0 0}, upload=19952]: Done begin upload segment. Dec 4 15:38:01.301883024 (22h42m37.773093753s) [noise=true , address=145.79.227.40:4051 , piece_id=RLDHK2A262X6IZQWP6DK2IPWBTMXYZ4YQQ7SG4IJRSO3KDPBUJYA, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.301945323 (22h42m37.773156092s) [noise=true , address=145.79.227.42:4121 , piece_id=USB7H5RUYH4JMFQQRO5M3BSKB4NETHDTSGV5CJRFOT76D32V6DHQ, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.302924858 (22h42m37.774135597s) [noise=true , address=145.79.227.23:4547 , piece_id=AGTUJHQKSNRQS6TR6Z6JL6HR47LYKLVQGY5WZR7Z4RBXC7NCLWYQ, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.303471950 (22h42m37.774682689s) [noise=true , address=145.79.227.34:4906 , piece_id=XNS7EVNDDK3XQVIYL5RPPIJPPTFNSTVJFWF6Q55EL3SRRAQTRYEQ, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.304017129 (22h42m37.775227868s) [noise=true , address=145.79.227.8:4330 , piece_id=3PCXR266AYJQSCQDCPBESLN7H4WLS6BLXDQWMWUOBS37SRVVKDCQ, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.304628322 (22h42m37.775839081s) [noise=true , address=77.90.31.11:4555 , piece_id=NDNU7HIDBMQGCP5HKQEGWAQCSENZQ4XPL5PDBLZSUVJNUMMKDKLA, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.305202977 (22h42m37.776413716s) [noise=true , address=145.79.227.27:4691 , piece_id=CIIRLVK6IODAMCRN7LBRLXQ5IR2AC7J3QAPHZUA3GROJIWNNEELQ, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.305742744 (22h42m37.776953483s) [noise=true , address=145.79.227.14:4840 , piece_id=7CFAXRK7F366A7QPI5OPTWMRO3Z5LK7LMEDP37AB5XLUMP34JVHQ, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.306643649 (22h42m37.777854389s) [noise=true , address=145.79.227.9:4367 , piece_id=E7UKP2XNQTELMOXMLN5CUTEG6ZVZE5TT4DW6ZYUD5733HRHWYD6Q, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.306719754 (22h42m37.777930493s) [noise=true , address=145.79.227.26:4655 , piece_id=O65JQUKSUPZSBOI5EPDDRSPMCN2LI2BYWBVDYSRIG4H6F2ASKS2A, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.307495471 (22h42m37.7787062s) [noise=true , address=145.79.227.7:4293 , piece_id=HYDMGVL463LIAO4N7JRFMFZSZ566C7DHHEMMVCPHTDUTEQYESKFQ, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.307859154 (22h42m37.779069883s) [noise=true , address=145.79.227.13:4804 , piece_id=ALPYEUHA2CMGWO6NFA7UBQUIJZTGE4NF7QYLNVAZQKYTJKABT6DA, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.308629862 (22h42m37.779840601s) [noise=true , address=145.79.227.16:4911 , piece_id=2M7N5623QL2BC6IMB3RJVRUBLB2SNZ66SWZY45N6WBYLCC56IZMA, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.310368402 (22h42m37.781579141s) [noise=true , address=145.79.227.21:4474 , piece_id=EQILGZ4H4R254DE7UW43GYCETN77D5FIRK4SE4Y36PPPSOKBSX5A, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.315008367 (22h42m37.786219106s) [noise=true , address=145.79.227.32:4834 , piece_id=CIJJAFXIJYJ6MHGVXTZ6FLU36QWE3HNVOLWW4IMUH5ZCWTL6ETZQ, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.317256889 (22h42m37.788467628s) [noise=true , address=145.79.227.31:4799 , piece_id=FBKOSDEM2H5ETZM6OOGLZCDGTT5RQSLMRZ2CLX2GOZRHRCMR2RNA, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:03.018370042 (22h42m39.489580781s) [upload=19952]: Next returned nil segment Dec 4 15:38:03.018386903 (22h42m39.489597642s) [upload=19952]: Waiting for error group managing segments... Dec 4 15:38:03.360528210 (22h42m39.831738939s) [noise=true , address=77.90.31.11:4555 , piece_id=NDNU7HIDBMQGCP5HKQEGWAQCSENZQ4XPL5PDBLZSUVJNUMMKDKLA, seg_pos={0 0}, upload=19952]: Done uploading piece. err: <nil> Dec 4 15:38:03.476269451 (22h42m39.94748018s) [noise=true , address=145.79.227.16:4911 , piece_id=2M7N5623QL2BC6IMB3RJVRUBLB2SNZ66SWZY45N6WBYLCC56IZMA, seg_pos={0 0}, upload=19952]: Done uploading piece. err: <nil> Dec 4 15:38:03.477241432 (22h42m39.948452171s) [noise=true , address=145.79.227.9:4367 , piece_id=E7UKP2XNQTELMOXMLN5CUTEG6ZVZE5TT4DW6ZYUD5733HRHWYD6Q, seg_pos={0 0}, upload=19952]: Done uploading piece. err: <nil> Dec 4 15:38:03.483785764 (22h42m39.954996503s) [noise=true , address=145.79.227.14:4840 , piece_id=7CFAXRK7F366A7QPI5OPTWMRO3Z5LK7LMEDP37AB5XLUMP34JVHQ, seg_pos={0 0}, upload=19952]: Done uploading piece. err: <nil> Dec 4 15:38:03.484258473 (22h42m39.955469212s) [noise=true , address=145.79.227.27:4691 , piece_id=CIIRLVK6IODAMCRN7LBRLXQ5IR2AC7J3QAPHZUA3GROJIWNNEELQ, seg_pos={0 0}, upload=19952]: Done uploading piece. err: <nil> Dec 4 15:38:03.487612530 (22h42m39.958823269s) [noise=true , address=145.79.227.23:4547 , piece_id=AGTUJHQKSNRQS6TR6Z6JL6HR47LYKLVQGY5WZR7Z4RBXC7NCLWYQ, seg_pos={0 0}, upload=19952]: Done uploading piece. err: <nil> Dec 4 15:38:03.495282255 (22h42m39.966492984s) [noise=true , address=145.79.227.40:4051 , piece_id=RLDHK2A262X6IZQWP6DK2IPWBTMXYZ4YQQ7SG4IJRSO3KDPBUJYA, seg_pos={0 0}, upload=19952]: Done uploading piece. err: <nil> Dec 4 15:38:03.509793507 (22h42m39.981004236s) [noise=true , address=145.79.227.8:4330 , piece_id=3PCXR266AYJQSCQDCPBESLN7H4WLS6BLXDQWMWUOBS37SRVVKDCQ, seg_pos={0 0}, upload=19952]: Done uploading piece. err: <nil> Dec 4 15:38:03.513640021 (22h42m39.98485076s) [noise=true , address=145.79.227.34:4906 , piece_id=XNS7EVNDDK3XQVIYL5RPPIJPPTFNSTVJFWF6Q55EL3SRRAQTRYEQ, seg_pos={0 0}, upload=19952]: Done uploading piece. err: <nil> Dec 4 15:38:03.785547720 (22h42m40.256758459s) [noise=true , address=145.79.227.13:4804 , piece_id=ALPYEUHA2CMGWO6NFA7UBQUIJZTGE4NF7QYLNVAZQKYTJKABT6DA, seg_pos={0 0}, upload=19952]: Done uploading piece. err: <nil> Dec 4 15:38:03.809124804 (22h42m40.280335543s) [noise=true , address=145.79.227.21:4474 , piece_id=EQILGZ4H4R254DE7UW43GYCETN77D5FIRK4SE4Y36PPPSOKBSX5A, seg_pos={0 0}, upload=19952]: Done uploading piece. err: <nil> Dec 4 15:38:04.138848766 (22h42m40.610059495s) [noise=true , address=145.79.227.7:4293 , piece_id=HYDMGVL463LIAO4N7JRFMFZSZ566C7DHHEMMVCPHTDUTEQYESKFQ, seg_pos={0 0}, upload=19952]: Done uploading piece. err: <nil> Dec 4 15:38:04.176120728 (22h42m40.647331457s) [noise=true , address=145.79.227.32:4834 , piece_id=CIJJAFXIJYJ6MHGVXTZ6FLU36QWE3HNVOLWW4IMUH5ZCWTL6ETZQ, seg_pos={0 0}, upload=19952]: Done uploading piece. err: <nil> Dec 4 15:38:04.176143252 (22h42m40.647353991s) [seg_pos={0 0}, upload=19952]: Segment reached optimal threshold of 13 pieces. Dec 4 15:38:04.176373089 (22h42m40.647583818s) [noise=true , address=145.79.227.26:4655 , piece_id=O65JQUKSUPZSBOI5EPDDRSPMCN2LI2BYWBVDYSRIG4H6F2ASKS2A, seg_pos={0 0}, upload=19952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xpZx5VbgHmbepaS9ZXgHfp7SZGeYPEjuJddQeYPUVpDU4EGx5): protocol: write tcp 188.166.225.170:46794->145.79.227.26:4655: use of closed network connection; write tcp 188.166.225.170:46794->145.79.227.26:4655: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46794->145.79.227.26:4655: use of closed network connection Dec 4 15:38:04.176382406 (22h42m40.647593145s) [noise=true , address=145.79.227.26:4655 , piece_id=O65JQUKSUPZSBOI5EPDDRSPMCN2LI2BYWBVDYSRIG4H6F2ASKS2A, seg_pos={0 0}, upload=19952]: Long tail cancellation due to optimal threshold Dec 4 15:38:04.176643925 (22h42m40.647854654s) [noise=true , address=145.79.227.31:4799 , piece_id=FBKOSDEM2H5ETZM6OOGLZCDGTT5RQSLMRZ2CLX2GOZRHRCMR2RNA, seg_pos={0 0}, upload=19952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X): protocol: expected piece hash; context canceled; EOF Dec 4 15:38:04.176651369 (22h42m40.647862108s) [noise=true , address=145.79.227.31:4799 , piece_id=FBKOSDEM2H5ETZM6OOGLZCDGTT5RQSLMRZ2CLX2GOZRHRCMR2RNA, seg_pos={0 0}, upload=19952]: Long tail cancellation due to optimal threshold Dec 4 15:38:04.176821292 (22h42m40.648032041s) [noise=true , address=145.79.227.42:4121 , piece_id=USB7H5RUYH4JMFQQRO5M3BSKB4NETHDTSGV5CJRFOT76D32V6DHQ, seg_pos={0 0}, upload=19952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh): protocol: expected piece hash; context canceled; EOF Dec 4 15:38:04.176869694 (22h42m40.648080433s) [noise=true , address=145.79.227.42:4121 , piece_id=USB7H5RUYH4JMFQQRO5M3BSKB4NETHDTSGV5CJRFOT76D32V6DHQ, seg_pos={0 0}, upload=19952]: Long tail cancellation due to optimal threshold Dec 4 15:38:04.176908829 (22h42m40.648119569s) [seg_pos={0 0}, upload=19952]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 4 15:38:04.177077430 (22h42m40.648288169s) [upload=19952]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 4 15:38:04.177087099 (22h42m40.648297848s) [upload=19952]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 4 15:38:04.502975285 (22h42m40.974186014s) [upload=19952]: Done uploading segments... Dec 4 15:38:04.502992749 (22h42m40.974203488s) [upload=19952]: Upload finished. err: <nil> Duration: 5.899126836s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 5.742553156s trace: https://jaeger.dev.storj.io/trace/1729cf59b626c72a FAIL (2025-12-04 16:09:38 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw, Piece ID: XNS7EVNDDK3XQVIYL5RPPIJPPTFNSTVJFWF6Q55EL3SRRAQTRYEQ) context deadline exceeded; piecestore: (Node ID: 12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, Piece ID: HYDMGVL463LIAO4N7JRFMFZSZ566C7DHHEMMVCPHTDUTEQYESKFQ) context deadline exceeded; piecestore: (Node ID: 1rL2DnjSgHP36Md3bCgy5padqTAhv3iQCeURbLMuuuMBxtUNZH, Piece ID: EQILGZ4H4R254DE7UW43GYCETN77D5FIRK4SE4Y36PPPSOKBSX5A) context deadline exceeded; piecestore: (Node ID: 12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n, Piece ID: RLDHK2A262X6IZQWP6DK2IPWBTMXYZ4YQQ7SG4IJRSO3KDPBUJYA) context deadline exceeded ======LOGS====== Dec 4 15:38:01.034643304 (22h42m37.505854043s) [upload=19952]: Starting upload Dec 4 15:38:01.034798069 (22h42m37.506008818s) [upload=19952]: Done starting upload Dec 4 15:38:01.034989493 (22h42m37.506200232s) [upload=19952]: Uploading segments... Dec 4 15:38:01.035032334 (22h42m37.506243073s) [upload=19952]: Got next segment. Inline: false Dec 4 15:38:01.035060508 (22h42m37.506271237s) [upload=19952]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 15:38:01.035064726 (22h42m37.506275455s) [upload=19952]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 15:38:01.301722830 (22h42m37.772933559s) [seg_pos={0 0}, upload=19952]: Begin upload segment... Dec 4 15:38:01.301843599 (22h42m37.773054338s) [seg_pos={0 0}, upload=19952]: Done begin upload segment. Dec 4 15:38:01.301883024 (22h42m37.773093753s) [noise=true , address=145.79.227.40:4051 , piece_id=RLDHK2A262X6IZQWP6DK2IPWBTMXYZ4YQQ7SG4IJRSO3KDPBUJYA, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.301945323 (22h42m37.773156092s) [noise=true , address=145.79.227.42:4121 , piece_id=USB7H5RUYH4JMFQQRO5M3BSKB4NETHDTSGV5CJRFOT76D32V6DHQ, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.302924858 (22h42m37.774135597s) [noise=true , address=145.79.227.23:4547 , piece_id=AGTUJHQKSNRQS6TR6Z6JL6HR47LYKLVQGY5WZR7Z4RBXC7NCLWYQ, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.303471950 (22h42m37.774682689s) [noise=true , address=145.79.227.34:4906 , piece_id=XNS7EVNDDK3XQVIYL5RPPIJPPTFNSTVJFWF6Q55EL3SRRAQTRYEQ, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.304017129 (22h42m37.775227868s) [noise=true , address=145.79.227.8:4330 , piece_id=3PCXR266AYJQSCQDCPBESLN7H4WLS6BLXDQWMWUOBS37SRVVKDCQ, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.304628322 (22h42m37.775839081s) [noise=true , address=77.90.31.11:4555 , piece_id=NDNU7HIDBMQGCP5HKQEGWAQCSENZQ4XPL5PDBLZSUVJNUMMKDKLA, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.305202977 (22h42m37.776413716s) [noise=true , address=145.79.227.27:4691 , piece_id=CIIRLVK6IODAMCRN7LBRLXQ5IR2AC7J3QAPHZUA3GROJIWNNEELQ, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.305742744 (22h42m37.776953483s) [noise=true , address=145.79.227.14:4840 , piece_id=7CFAXRK7F366A7QPI5OPTWMRO3Z5LK7LMEDP37AB5XLUMP34JVHQ, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.306643649 (22h42m37.777854389s) [noise=true , address=145.79.227.9:4367 , piece_id=E7UKP2XNQTELMOXMLN5CUTEG6ZVZE5TT4DW6ZYUD5733HRHWYD6Q, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.306719754 (22h42m37.777930493s) [noise=true , address=145.79.227.26:4655 , piece_id=O65JQUKSUPZSBOI5EPDDRSPMCN2LI2BYWBVDYSRIG4H6F2ASKS2A, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.307495471 (22h42m37.7787062s) [noise=true , address=145.79.227.7:4293 , piece_id=HYDMGVL463LIAO4N7JRFMFZSZ566C7DHHEMMVCPHTDUTEQYESKFQ, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.307859154 (22h42m37.779069883s) [noise=true , address=145.79.227.13:4804 , piece_id=ALPYEUHA2CMGWO6NFA7UBQUIJZTGE4NF7QYLNVAZQKYTJKABT6DA, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.308629862 (22h42m37.779840601s) [noise=true , address=145.79.227.16:4911 , piece_id=2M7N5623QL2BC6IMB3RJVRUBLB2SNZ66SWZY45N6WBYLCC56IZMA, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.310368402 (22h42m37.781579141s) [noise=true , address=145.79.227.21:4474 , piece_id=EQILGZ4H4R254DE7UW43GYCETN77D5FIRK4SE4Y36PPPSOKBSX5A, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.315008367 (22h42m37.786219106s) [noise=true , address=145.79.227.32:4834 , piece_id=CIJJAFXIJYJ6MHGVXTZ6FLU36QWE3HNVOLWW4IMUH5ZCWTL6ETZQ, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.317256889 (22h42m37.788467628s) [noise=true , address=145.79.227.31:4799 , piece_id=FBKOSDEM2H5ETZM6OOGLZCDGTT5RQSLMRZ2CLX2GOZRHRCMR2RNA, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:03.018370042 (22h42m39.489580781s) [upload=19952]: Next returned nil segment Dec 4 15:38:03.018386903 (22h42m39.489597642s) [upload=19952]: Waiting for error group managing segments... Dec 4 15:38:03.360528210 (22h42m39.831738939s) [noise=true , address=77.90.31.11:4555 , piece_id=NDNU7HIDBMQGCP5HKQEGWAQCSENZQ4XPL5PDBLZSUVJNUMMKDKLA, seg_pos={0 0}, upload=19952]: Done uploading piece. err: <nil> Dec 4 15:38:03.476269451 (22h42m39.94748018s) [noise=true , address=145.79.227.16:4911 , piece_id=2M7N5623QL2BC6IMB3RJVRUBLB2SNZ66SWZY45N6WBYLCC56IZMA, seg_pos={0 0}, upload=19952]: Done uploading piece. err: <nil> Dec 4 15:38:03.477241432 (22h42m39.948452171s) [noise=true , address=145.79.227.9:4367 , piece_id=E7UKP2XNQTELMOXMLN5CUTEG6ZVZE5TT4DW6ZYUD5733HRHWYD6Q, seg_pos={0 0}, upload=19952]: Done uploading piece. err: <nil> Dec 4 15:38:03.483785764 (22h42m39.954996503s) [noise=true , address=145.79.227.14:4840 , piece_id=7CFAXRK7F366A7QPI5OPTWMRO3Z5LK7LMEDP37AB5XLUMP34JVHQ, seg_pos={0 0}, upload=19952]: Done uploading piece. err: <nil> Dec 4 15:38:03.484258473 (22h42m39.955469212s) [noise=true , address=145.79.227.27:4691 , piece_id=CIIRLVK6IODAMCRN7LBRLXQ5IR2AC7J3QAPHZUA3GROJIWNNEELQ, seg_pos={0 0}, upload=19952]: Done uploading piece. err: <nil> Dec 4 15:38:03.487612530 (22h42m39.958823269s) [noise=true , address=145.79.227.23:4547 , piece_id=AGTUJHQKSNRQS6TR6Z6JL6HR47LYKLVQGY5WZR7Z4RBXC7NCLWYQ, seg_pos={0 0}, upload=19952]: Done uploading piece. err: <nil> Dec 4 15:38:03.495282255 (22h42m39.966492984s) [noise=true , address=145.79.227.40:4051 , piece_id=RLDHK2A262X6IZQWP6DK2IPWBTMXYZ4YQQ7SG4IJRSO3KDPBUJYA, seg_pos={0 0}, upload=19952]: Done uploading piece. err: <nil> Dec 4 15:38:03.509793507 (22h42m39.981004236s) [noise=true , address=145.79.227.8:4330 , piece_id=3PCXR266AYJQSCQDCPBESLN7H4WLS6BLXDQWMWUOBS37SRVVKDCQ, seg_pos={0 0}, upload=19952]: Done uploading piece. err: <nil> Dec 4 15:38:03.513640021 (22h42m39.98485076s) [noise=true , address=145.79.227.34:4906 , piece_id=XNS7EVNDDK3XQVIYL5RPPIJPPTFNSTVJFWF6Q55EL3SRRAQTRYEQ, seg_pos={0 0}, upload=19952]: Done uploading piece. err: <nil> Dec 4 15:38:03.785547720 (22h42m40.256758459s) [noise=true , address=145.79.227.13:4804 , piece_id=ALPYEUHA2CMGWO6NFA7UBQUIJZTGE4NF7QYLNVAZQKYTJKABT6DA, seg_pos={0 0}, upload=19952]: Done uploading piece. err: <nil> Dec 4 15:38:03.809124804 (22h42m40.280335543s) [noise=true , address=145.79.227.21:4474 , piece_id=EQILGZ4H4R254DE7UW43GYCETN77D5FIRK4SE4Y36PPPSOKBSX5A, seg_pos={0 0}, upload=19952]: Done uploading piece. err: <nil> Dec 4 15:38:04.138848766 (22h42m40.610059495s) [noise=true , address=145.79.227.7:4293 , piece_id=HYDMGVL463LIAO4N7JRFMFZSZ566C7DHHEMMVCPHTDUTEQYESKFQ, seg_pos={0 0}, upload=19952]: Done uploading piece. err: <nil> Dec 4 15:38:04.176120728 (22h42m40.647331457s) [noise=true , address=145.79.227.32:4834 , piece_id=CIJJAFXIJYJ6MHGVXTZ6FLU36QWE3HNVOLWW4IMUH5ZCWTL6ETZQ, seg_pos={0 0}, upload=19952]: Done uploading piece. err: <nil> Dec 4 15:38:04.176143252 (22h42m40.647353991s) [seg_pos={0 0}, upload=19952]: Segment reached optimal threshold of 13 pieces. Dec 4 15:38:04.176373089 (22h42m40.647583818s) [noise=true , address=145.79.227.26:4655 , piece_id=O65JQUKSUPZSBOI5EPDDRSPMCN2LI2BYWBVDYSRIG4H6F2ASKS2A, seg_pos={0 0}, upload=19952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xpZx5VbgHmbepaS9ZXgHfp7SZGeYPEjuJddQeYPUVpDU4EGx5): protocol: write tcp 188.166.225.170:46794->145.79.227.26:4655: use of closed network connection; write tcp 188.166.225.170:46794->145.79.227.26:4655: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46794->145.79.227.26:4655: use of closed network connection Dec 4 15:38:04.176382406 (22h42m40.647593145s) [noise=true , address=145.79.227.26:4655 , piece_id=O65JQUKSUPZSBOI5EPDDRSPMCN2LI2BYWBVDYSRIG4H6F2ASKS2A, seg_pos={0 0}, upload=19952]: Long tail cancellation due to optimal threshold Dec 4 15:38:04.176643925 (22h42m40.647854654s) [noise=true , address=145.79.227.31:4799 , piece_id=FBKOSDEM2H5ETZM6OOGLZCDGTT5RQSLMRZ2CLX2GOZRHRCMR2RNA, seg_pos={0 0}, upload=19952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X): protocol: expected piece hash; context canceled; EOF Dec 4 15:38:04.176651369 (22h42m40.647862108s) [noise=true , address=145.79.227.31:4799 , piece_id=FBKOSDEM2H5ETZM6OOGLZCDGTT5RQSLMRZ2CLX2GOZRHRCMR2RNA, seg_pos={0 0}, upload=19952]: Long tail cancellation due to optimal threshold Dec 4 15:38:04.176821292 (22h42m40.648032041s) [noise=true , address=145.79.227.42:4121 , piece_id=USB7H5RUYH4JMFQQRO5M3BSKB4NETHDTSGV5CJRFOT76D32V6DHQ, seg_pos={0 0}, upload=19952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh): protocol: expected piece hash; context canceled; EOF Dec 4 15:38:04.176869694 (22h42m40.648080433s) [noise=true , address=145.79.227.42:4121 , piece_id=USB7H5RUYH4JMFQQRO5M3BSKB4NETHDTSGV5CJRFOT76D32V6DHQ, seg_pos={0 0}, upload=19952]: Long tail cancellation due to optimal threshold Dec 4 15:38:04.176908829 (22h42m40.648119569s) [seg_pos={0 0}, upload=19952]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 4 15:38:04.177077430 (22h42m40.648288169s) [upload=19952]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 4 15:38:04.177087099 (22h42m40.648297848s) [upload=19952]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 4 15:38:04.502975285 (22h42m40.974186014s) [upload=19952]: Done uploading segments... Dec 4 15:38:04.502992749 (22h42m40.974203488s) [upload=19952]: Upload finished. err: <nil> Duration: 5.74260169s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 5.965652652s trace: https://jaeger.dev.storj.io/trace/1729cf59b626c72a FAIL (2025-12-04 16:09:14 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw, Piece ID: XNS7EVNDDK3XQVIYL5RPPIJPPTFNSTVJFWF6Q55EL3SRRAQTRYEQ) context deadline exceeded; piecestore: (Node ID: 12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, Piece ID: HYDMGVL463LIAO4N7JRFMFZSZ566C7DHHEMMVCPHTDUTEQYESKFQ) context deadline exceeded; piecestore: (Node ID: 1rL2DnjSgHP36Md3bCgy5padqTAhv3iQCeURbLMuuuMBxtUNZH, Piece ID: EQILGZ4H4R254DE7UW43GYCETN77D5FIRK4SE4Y36PPPSOKBSX5A) context deadline exceeded; piecestore: (Node ID: 12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n, Piece ID: RLDHK2A262X6IZQWP6DK2IPWBTMXYZ4YQQ7SG4IJRSO3KDPBUJYA) context deadline exceeded ======LOGS====== Dec 4 15:38:01.034643304 (22h42m37.505854043s) [upload=19952]: Starting upload Dec 4 15:38:01.034798069 (22h42m37.506008818s) [upload=19952]: Done starting upload Dec 4 15:38:01.034989493 (22h42m37.506200232s) [upload=19952]: Uploading segments... Dec 4 15:38:01.035032334 (22h42m37.506243073s) [upload=19952]: Got next segment. Inline: false Dec 4 15:38:01.035060508 (22h42m37.506271237s) [upload=19952]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 15:38:01.035064726 (22h42m37.506275455s) [upload=19952]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 15:38:01.301722830 (22h42m37.772933559s) [seg_pos={0 0}, upload=19952]: Begin upload segment... Dec 4 15:38:01.301843599 (22h42m37.773054338s) [seg_pos={0 0}, upload=19952]: Done begin upload segment. Dec 4 15:38:01.301883024 (22h42m37.773093753s) [noise=true , address=145.79.227.40:4051 , piece_id=RLDHK2A262X6IZQWP6DK2IPWBTMXYZ4YQQ7SG4IJRSO3KDPBUJYA, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.301945323 (22h42m37.773156092s) [noise=true , address=145.79.227.42:4121 , piece_id=USB7H5RUYH4JMFQQRO5M3BSKB4NETHDTSGV5CJRFOT76D32V6DHQ, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.302924858 (22h42m37.774135597s) [noise=true , address=145.79.227.23:4547 , piece_id=AGTUJHQKSNRQS6TR6Z6JL6HR47LYKLVQGY5WZR7Z4RBXC7NCLWYQ, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.303471950 (22h42m37.774682689s) [noise=true , address=145.79.227.34:4906 , piece_id=XNS7EVNDDK3XQVIYL5RPPIJPPTFNSTVJFWF6Q55EL3SRRAQTRYEQ, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.304017129 (22h42m37.775227868s) [noise=true , address=145.79.227.8:4330 , piece_id=3PCXR266AYJQSCQDCPBESLN7H4WLS6BLXDQWMWUOBS37SRVVKDCQ, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.304628322 (22h42m37.775839081s) [noise=true , address=77.90.31.11:4555 , piece_id=NDNU7HIDBMQGCP5HKQEGWAQCSENZQ4XPL5PDBLZSUVJNUMMKDKLA, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.305202977 (22h42m37.776413716s) [noise=true , address=145.79.227.27:4691 , piece_id=CIIRLVK6IODAMCRN7LBRLXQ5IR2AC7J3QAPHZUA3GROJIWNNEELQ, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.305742744 (22h42m37.776953483s) [noise=true , address=145.79.227.14:4840 , piece_id=7CFAXRK7F366A7QPI5OPTWMRO3Z5LK7LMEDP37AB5XLUMP34JVHQ, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.306643649 (22h42m37.777854389s) [noise=true , address=145.79.227.9:4367 , piece_id=E7UKP2XNQTELMOXMLN5CUTEG6ZVZE5TT4DW6ZYUD5733HRHWYD6Q, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.306719754 (22h42m37.777930493s) [noise=true , address=145.79.227.26:4655 , piece_id=O65JQUKSUPZSBOI5EPDDRSPMCN2LI2BYWBVDYSRIG4H6F2ASKS2A, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.307495471 (22h42m37.7787062s) [noise=true , address=145.79.227.7:4293 , piece_id=HYDMGVL463LIAO4N7JRFMFZSZ566C7DHHEMMVCPHTDUTEQYESKFQ, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.307859154 (22h42m37.779069883s) [noise=true , address=145.79.227.13:4804 , piece_id=ALPYEUHA2CMGWO6NFA7UBQUIJZTGE4NF7QYLNVAZQKYTJKABT6DA, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.308629862 (22h42m37.779840601s) [noise=true , address=145.79.227.16:4911 , piece_id=2M7N5623QL2BC6IMB3RJVRUBLB2SNZ66SWZY45N6WBYLCC56IZMA, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.310368402 (22h42m37.781579141s) [noise=true , address=145.79.227.21:4474 , piece_id=EQILGZ4H4R254DE7UW43GYCETN77D5FIRK4SE4Y36PPPSOKBSX5A, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.315008367 (22h42m37.786219106s) [noise=true , address=145.79.227.32:4834 , piece_id=CIJJAFXIJYJ6MHGVXTZ6FLU36QWE3HNVOLWW4IMUH5ZCWTL6ETZQ, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.317256889 (22h42m37.788467628s) [noise=true , address=145.79.227.31:4799 , piece_id=FBKOSDEM2H5ETZM6OOGLZCDGTT5RQSLMRZ2CLX2GOZRHRCMR2RNA, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:03.018370042 (22h42m39.489580781s) [upload=19952]: Next returned nil segment Dec 4 15:38:03.018386903 (22h42m39.489597642s) [upload=19952]: Waiting for error group managing segments... Dec 4 15:38:03.360528210 (22h42m39.831738939s) [noise=true , address=77.90.31.11:4555 , piece_id=NDNU7HIDBMQGCP5HKQEGWAQCSENZQ4XPL5PDBLZSUVJNUMMKDKLA, seg_pos={0 0}, upload=19952]: Done uploading piece. err: <nil> Dec 4 15:38:03.476269451 (22h42m39.94748018s) [noise=true , address=145.79.227.16:4911 , piece_id=2M7N5623QL2BC6IMB3RJVRUBLB2SNZ66SWZY45N6WBYLCC56IZMA, seg_pos={0 0}, upload=19952]: Done uploading piece. err: <nil> Dec 4 15:38:03.477241432 (22h42m39.948452171s) [noise=true , address=145.79.227.9:4367 , piece_id=E7UKP2XNQTELMOXMLN5CUTEG6ZVZE5TT4DW6ZYUD5733HRHWYD6Q, seg_pos={0 0}, upload=19952]: Done uploading piece. err: <nil> Dec 4 15:38:03.483785764 (22h42m39.954996503s) [noise=true , address=145.79.227.14:4840 , piece_id=7CFAXRK7F366A7QPI5OPTWMRO3Z5LK7LMEDP37AB5XLUMP34JVHQ, seg_pos={0 0}, upload=19952]: Done uploading piece. err: <nil> Dec 4 15:38:03.484258473 (22h42m39.955469212s) [noise=true , address=145.79.227.27:4691 , piece_id=CIIRLVK6IODAMCRN7LBRLXQ5IR2AC7J3QAPHZUA3GROJIWNNEELQ, seg_pos={0 0}, upload=19952]: Done uploading piece. err: <nil> Dec 4 15:38:03.487612530 (22h42m39.958823269s) [noise=true , address=145.79.227.23:4547 , piece_id=AGTUJHQKSNRQS6TR6Z6JL6HR47LYKLVQGY5WZR7Z4RBXC7NCLWYQ, seg_pos={0 0}, upload=19952]: Done uploading piece. err: <nil> Dec 4 15:38:03.495282255 (22h42m39.966492984s) [noise=true , address=145.79.227.40:4051 , piece_id=RLDHK2A262X6IZQWP6DK2IPWBTMXYZ4YQQ7SG4IJRSO3KDPBUJYA, seg_pos={0 0}, upload=19952]: Done uploading piece. err: <nil> Dec 4 15:38:03.509793507 (22h42m39.981004236s) [noise=true , address=145.79.227.8:4330 , piece_id=3PCXR266AYJQSCQDCPBESLN7H4WLS6BLXDQWMWUOBS37SRVVKDCQ, seg_pos={0 0}, upload=19952]: Done uploading piece. err: <nil> Dec 4 15:38:03.513640021 (22h42m39.98485076s) [noise=true , address=145.79.227.34:4906 , piece_id=XNS7EVNDDK3XQVIYL5RPPIJPPTFNSTVJFWF6Q55EL3SRRAQTRYEQ, seg_pos={0 0}, upload=19952]: Done uploading piece. err: <nil> Dec 4 15:38:03.785547720 (22h42m40.256758459s) [noise=true , address=145.79.227.13:4804 , piece_id=ALPYEUHA2CMGWO6NFA7UBQUIJZTGE4NF7QYLNVAZQKYTJKABT6DA, seg_pos={0 0}, upload=19952]: Done uploading piece. err: <nil> Dec 4 15:38:03.809124804 (22h42m40.280335543s) [noise=true , address=145.79.227.21:4474 , piece_id=EQILGZ4H4R254DE7UW43GYCETN77D5FIRK4SE4Y36PPPSOKBSX5A, seg_pos={0 0}, upload=19952]: Done uploading piece. err: <nil> Dec 4 15:38:04.138848766 (22h42m40.610059495s) [noise=true , address=145.79.227.7:4293 , piece_id=HYDMGVL463LIAO4N7JRFMFZSZ566C7DHHEMMVCPHTDUTEQYESKFQ, seg_pos={0 0}, upload=19952]: Done uploading piece. err: <nil> Dec 4 15:38:04.176120728 (22h42m40.647331457s) [noise=true , address=145.79.227.32:4834 , piece_id=CIJJAFXIJYJ6MHGVXTZ6FLU36QWE3HNVOLWW4IMUH5ZCWTL6ETZQ, seg_pos={0 0}, upload=19952]: Done uploading piece. err: <nil> Dec 4 15:38:04.176143252 (22h42m40.647353991s) [seg_pos={0 0}, upload=19952]: Segment reached optimal threshold of 13 pieces. Dec 4 15:38:04.176373089 (22h42m40.647583818s) [noise=true , address=145.79.227.26:4655 , piece_id=O65JQUKSUPZSBOI5EPDDRSPMCN2LI2BYWBVDYSRIG4H6F2ASKS2A, seg_pos={0 0}, upload=19952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xpZx5VbgHmbepaS9ZXgHfp7SZGeYPEjuJddQeYPUVpDU4EGx5): protocol: write tcp 188.166.225.170:46794->145.79.227.26:4655: use of closed network connection; write tcp 188.166.225.170:46794->145.79.227.26:4655: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46794->145.79.227.26:4655: use of closed network connection Dec 4 15:38:04.176382406 (22h42m40.647593145s) [noise=true , address=145.79.227.26:4655 , piece_id=O65JQUKSUPZSBOI5EPDDRSPMCN2LI2BYWBVDYSRIG4H6F2ASKS2A, seg_pos={0 0}, upload=19952]: Long tail cancellation due to optimal threshold Dec 4 15:38:04.176643925 (22h42m40.647854654s) [noise=true , address=145.79.227.31:4799 , piece_id=FBKOSDEM2H5ETZM6OOGLZCDGTT5RQSLMRZ2CLX2GOZRHRCMR2RNA, seg_pos={0 0}, upload=19952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X): protocol: expected piece hash; context canceled; EOF Dec 4 15:38:04.176651369 (22h42m40.647862108s) [noise=true , address=145.79.227.31:4799 , piece_id=FBKOSDEM2H5ETZM6OOGLZCDGTT5RQSLMRZ2CLX2GOZRHRCMR2RNA, seg_pos={0 0}, upload=19952]: Long tail cancellation due to optimal threshold Dec 4 15:38:04.176821292 (22h42m40.648032041s) [noise=true , address=145.79.227.42:4121 , piece_id=USB7H5RUYH4JMFQQRO5M3BSKB4NETHDTSGV5CJRFOT76D32V6DHQ, seg_pos={0 0}, upload=19952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh): protocol: expected piece hash; context canceled; EOF Dec 4 15:38:04.176869694 (22h42m40.648080433s) [noise=true , address=145.79.227.42:4121 , piece_id=USB7H5RUYH4JMFQQRO5M3BSKB4NETHDTSGV5CJRFOT76D32V6DHQ, seg_pos={0 0}, upload=19952]: Long tail cancellation due to optimal threshold Dec 4 15:38:04.176908829 (22h42m40.648119569s) [seg_pos={0 0}, upload=19952]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 4 15:38:04.177077430 (22h42m40.648288169s) [upload=19952]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 4 15:38:04.177087099 (22h42m40.648297848s) [upload=19952]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 4 15:38:04.502975285 (22h42m40.974186014s) [upload=19952]: Done uploading segments... Dec 4 15:38:04.502992749 (22h42m40.974203488s) [upload=19952]: Upload finished. err: <nil> Duration: 5.965678391s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 5.704509475s trace: https://jaeger.dev.storj.io/trace/1729cf59b626c72a FAIL (2025-12-04 16:08:58 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw, Piece ID: XNS7EVNDDK3XQVIYL5RPPIJPPTFNSTVJFWF6Q55EL3SRRAQTRYEQ) context deadline exceeded; piecestore: (Node ID: 12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, Piece ID: HYDMGVL463LIAO4N7JRFMFZSZ566C7DHHEMMVCPHTDUTEQYESKFQ) context deadline exceeded; piecestore: (Node ID: 1rL2DnjSgHP36Md3bCgy5padqTAhv3iQCeURbLMuuuMBxtUNZH, Piece ID: EQILGZ4H4R254DE7UW43GYCETN77D5FIRK4SE4Y36PPPSOKBSX5A) context deadline exceeded; piecestore: (Node ID: 12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n, Piece ID: RLDHK2A262X6IZQWP6DK2IPWBTMXYZ4YQQ7SG4IJRSO3KDPBUJYA) context deadline exceeded ======LOGS====== Dec 4 15:38:01.034643304 (22h42m37.505854043s) [upload=19952]: Starting upload Dec 4 15:38:01.034798069 (22h42m37.506008818s) [upload=19952]: Done starting upload Dec 4 15:38:01.034989493 (22h42m37.506200232s) [upload=19952]: Uploading segments... Dec 4 15:38:01.035032334 (22h42m37.506243073s) [upload=19952]: Got next segment. Inline: false Dec 4 15:38:01.035060508 (22h42m37.506271237s) [upload=19952]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 15:38:01.035064726 (22h42m37.506275455s) [upload=19952]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 15:38:01.301722830 (22h42m37.772933559s) [seg_pos={0 0}, upload=19952]: Begin upload segment... Dec 4 15:38:01.301843599 (22h42m37.773054338s) [seg_pos={0 0}, upload=19952]: Done begin upload segment. Dec 4 15:38:01.301883024 (22h42m37.773093753s) [noise=true , address=145.79.227.40:4051 , piece_id=RLDHK2A262X6IZQWP6DK2IPWBTMXYZ4YQQ7SG4IJRSO3KDPBUJYA, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.301945323 (22h42m37.773156092s) [noise=true , address=145.79.227.42:4121 , piece_id=USB7H5RUYH4JMFQQRO5M3BSKB4NETHDTSGV5CJRFOT76D32V6DHQ, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.302924858 (22h42m37.774135597s) [noise=true , address=145.79.227.23:4547 , piece_id=AGTUJHQKSNRQS6TR6Z6JL6HR47LYKLVQGY5WZR7Z4RBXC7NCLWYQ, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.303471950 (22h42m37.774682689s) [noise=true , address=145.79.227.34:4906 , piece_id=XNS7EVNDDK3XQVIYL5RPPIJPPTFNSTVJFWF6Q55EL3SRRAQTRYEQ, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.304017129 (22h42m37.775227868s) [noise=true , address=145.79.227.8:4330 , piece_id=3PCXR266AYJQSCQDCPBESLN7H4WLS6BLXDQWMWUOBS37SRVVKDCQ, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.304628322 (22h42m37.775839081s) [noise=true , address=77.90.31.11:4555 , piece_id=NDNU7HIDBMQGCP5HKQEGWAQCSENZQ4XPL5PDBLZSUVJNUMMKDKLA, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.305202977 (22h42m37.776413716s) [noise=true , address=145.79.227.27:4691 , piece_id=CIIRLVK6IODAMCRN7LBRLXQ5IR2AC7J3QAPHZUA3GROJIWNNEELQ, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.305742744 (22h42m37.776953483s) [noise=true , address=145.79.227.14:4840 , piece_id=7CFAXRK7F366A7QPI5OPTWMRO3Z5LK7LMEDP37AB5XLUMP34JVHQ, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.306643649 (22h42m37.777854389s) [noise=true , address=145.79.227.9:4367 , piece_id=E7UKP2XNQTELMOXMLN5CUTEG6ZVZE5TT4DW6ZYUD5733HRHWYD6Q, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.306719754 (22h42m37.777930493s) [noise=true , address=145.79.227.26:4655 , piece_id=O65JQUKSUPZSBOI5EPDDRSPMCN2LI2BYWBVDYSRIG4H6F2ASKS2A, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.307495471 (22h42m37.7787062s) [noise=true , address=145.79.227.7:4293 , piece_id=HYDMGVL463LIAO4N7JRFMFZSZ566C7DHHEMMVCPHTDUTEQYESKFQ, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.307859154 (22h42m37.779069883s) [noise=true , address=145.79.227.13:4804 , piece_id=ALPYEUHA2CMGWO6NFA7UBQUIJZTGE4NF7QYLNVAZQKYTJKABT6DA, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.308629862 (22h42m37.779840601s) [noise=true , address=145.79.227.16:4911 , piece_id=2M7N5623QL2BC6IMB3RJVRUBLB2SNZ66SWZY45N6WBYLCC56IZMA, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.310368402 (22h42m37.781579141s) [noise=true , address=145.79.227.21:4474 , piece_id=EQILGZ4H4R254DE7UW43GYCETN77D5FIRK4SE4Y36PPPSOKBSX5A, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.315008367 (22h42m37.786219106s) [noise=true , address=145.79.227.32:4834 , piece_id=CIJJAFXIJYJ6MHGVXTZ6FLU36QWE3HNVOLWW4IMUH5ZCWTL6ETZQ, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.317256889 (22h42m37.788467628s) [noise=true , address=145.79.227.31:4799 , piece_id=FBKOSDEM2H5ETZM6OOGLZCDGTT5RQSLMRZ2CLX2GOZRHRCMR2RNA, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:03.018370042 (22h42m39.489580781s) [upload=19952]: Next returned nil segment Dec 4 15:38:03.018386903 (22h42m39.489597642s) [upload=19952]: Waiting for error group managing segments... Dec 4 15:38:03.360528210 (22h42m39.831738939s) [noise=true , address=77.90.31.11:4555 , piece_id=NDNU7HIDBMQGCP5HKQEGWAQCSENZQ4XPL5PDBLZSUVJNUMMKDKLA, seg_pos={0 0}, upload=19952]: Done uploading piece. err: <nil> Dec 4 15:38:03.476269451 (22h42m39.94748018s) [noise=true , address=145.79.227.16:4911 , piece_id=2M7N5623QL2BC6IMB3RJVRUBLB2SNZ66SWZY45N6WBYLCC56IZMA, seg_pos={0 0}, upload=19952]: Done uploading piece. err: <nil> Dec 4 15:38:03.477241432 (22h42m39.948452171s) [noise=true , address=145.79.227.9:4367 , piece_id=E7UKP2XNQTELMOXMLN5CUTEG6ZVZE5TT4DW6ZYUD5733HRHWYD6Q, seg_pos={0 0}, upload=19952]: Done uploading piece. err: <nil> Dec 4 15:38:03.483785764 (22h42m39.954996503s) [noise=true , address=145.79.227.14:4840 , piece_id=7CFAXRK7F366A7QPI5OPTWMRO3Z5LK7LMEDP37AB5XLUMP34JVHQ, seg_pos={0 0}, upload=19952]: Done uploading piece. err: <nil> Dec 4 15:38:03.484258473 (22h42m39.955469212s) [noise=true , address=145.79.227.27:4691 , piece_id=CIIRLVK6IODAMCRN7LBRLXQ5IR2AC7J3QAPHZUA3GROJIWNNEELQ, seg_pos={0 0}, upload=19952]: Done uploading piece. err: <nil> Dec 4 15:38:03.487612530 (22h42m39.958823269s) [noise=true , address=145.79.227.23:4547 , piece_id=AGTUJHQKSNRQS6TR6Z6JL6HR47LYKLVQGY5WZR7Z4RBXC7NCLWYQ, seg_pos={0 0}, upload=19952]: Done uploading piece. err: <nil> Dec 4 15:38:03.495282255 (22h42m39.966492984s) [noise=true , address=145.79.227.40:4051 , piece_id=RLDHK2A262X6IZQWP6DK2IPWBTMXYZ4YQQ7SG4IJRSO3KDPBUJYA, seg_pos={0 0}, upload=19952]: Done uploading piece. err: <nil> Dec 4 15:38:03.509793507 (22h42m39.981004236s) [noise=true , address=145.79.227.8:4330 , piece_id=3PCXR266AYJQSCQDCPBESLN7H4WLS6BLXDQWMWUOBS37SRVVKDCQ, seg_pos={0 0}, upload=19952]: Done uploading piece. err: <nil> Dec 4 15:38:03.513640021 (22h42m39.98485076s) [noise=true , address=145.79.227.34:4906 , piece_id=XNS7EVNDDK3XQVIYL5RPPIJPPTFNSTVJFWF6Q55EL3SRRAQTRYEQ, seg_pos={0 0}, upload=19952]: Done uploading piece. err: <nil> Dec 4 15:38:03.785547720 (22h42m40.256758459s) [noise=true , address=145.79.227.13:4804 , piece_id=ALPYEUHA2CMGWO6NFA7UBQUIJZTGE4NF7QYLNVAZQKYTJKABT6DA, seg_pos={0 0}, upload=19952]: Done uploading piece. err: <nil> Dec 4 15:38:03.809124804 (22h42m40.280335543s) [noise=true , address=145.79.227.21:4474 , piece_id=EQILGZ4H4R254DE7UW43GYCETN77D5FIRK4SE4Y36PPPSOKBSX5A, seg_pos={0 0}, upload=19952]: Done uploading piece. err: <nil> Dec 4 15:38:04.138848766 (22h42m40.610059495s) [noise=true , address=145.79.227.7:4293 , piece_id=HYDMGVL463LIAO4N7JRFMFZSZ566C7DHHEMMVCPHTDUTEQYESKFQ, seg_pos={0 0}, upload=19952]: Done uploading piece. err: <nil> Dec 4 15:38:04.176120728 (22h42m40.647331457s) [noise=true , address=145.79.227.32:4834 , piece_id=CIJJAFXIJYJ6MHGVXTZ6FLU36QWE3HNVOLWW4IMUH5ZCWTL6ETZQ, seg_pos={0 0}, upload=19952]: Done uploading piece. err: <nil> Dec 4 15:38:04.176143252 (22h42m40.647353991s) [seg_pos={0 0}, upload=19952]: Segment reached optimal threshold of 13 pieces. Dec 4 15:38:04.176373089 (22h42m40.647583818s) [noise=true , address=145.79.227.26:4655 , piece_id=O65JQUKSUPZSBOI5EPDDRSPMCN2LI2BYWBVDYSRIG4H6F2ASKS2A, seg_pos={0 0}, upload=19952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xpZx5VbgHmbepaS9ZXgHfp7SZGeYPEjuJddQeYPUVpDU4EGx5): protocol: write tcp 188.166.225.170:46794->145.79.227.26:4655: use of closed network connection; write tcp 188.166.225.170:46794->145.79.227.26:4655: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46794->145.79.227.26:4655: use of closed network connection Dec 4 15:38:04.176382406 (22h42m40.647593145s) [noise=true , address=145.79.227.26:4655 , piece_id=O65JQUKSUPZSBOI5EPDDRSPMCN2LI2BYWBVDYSRIG4H6F2ASKS2A, seg_pos={0 0}, upload=19952]: Long tail cancellation due to optimal threshold Dec 4 15:38:04.176643925 (22h42m40.647854654s) [noise=true , address=145.79.227.31:4799 , piece_id=FBKOSDEM2H5ETZM6OOGLZCDGTT5RQSLMRZ2CLX2GOZRHRCMR2RNA, seg_pos={0 0}, upload=19952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X): protocol: expected piece hash; context canceled; EOF Dec 4 15:38:04.176651369 (22h42m40.647862108s) [noise=true , address=145.79.227.31:4799 , piece_id=FBKOSDEM2H5ETZM6OOGLZCDGTT5RQSLMRZ2CLX2GOZRHRCMR2RNA, seg_pos={0 0}, upload=19952]: Long tail cancellation due to optimal threshold Dec 4 15:38:04.176821292 (22h42m40.648032041s) [noise=true , address=145.79.227.42:4121 , piece_id=USB7H5RUYH4JMFQQRO5M3BSKB4NETHDTSGV5CJRFOT76D32V6DHQ, seg_pos={0 0}, upload=19952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh): protocol: expected piece hash; context canceled; EOF Dec 4 15:38:04.176869694 (22h42m40.648080433s) [noise=true , address=145.79.227.42:4121 , piece_id=USB7H5RUYH4JMFQQRO5M3BSKB4NETHDTSGV5CJRFOT76D32V6DHQ, seg_pos={0 0}, upload=19952]: Long tail cancellation due to optimal threshold Dec 4 15:38:04.176908829 (22h42m40.648119569s) [seg_pos={0 0}, upload=19952]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 4 15:38:04.177077430 (22h42m40.648288169s) [upload=19952]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 4 15:38:04.177087099 (22h42m40.648297848s) [upload=19952]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 4 15:38:04.502975285 (22h42m40.974186014s) [upload=19952]: Done uploading segments... Dec 4 15:38:04.502992749 (22h42m40.974203488s) [upload=19952]: Upload finished. err: <nil> Duration: 5.70453871s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 5.705418484s trace: https://jaeger.dev.storj.io/trace/1729cf59b626c72a FAIL (2025-12-04 16:08:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw, Piece ID: XNS7EVNDDK3XQVIYL5RPPIJPPTFNSTVJFWF6Q55EL3SRRAQTRYEQ) context deadline exceeded; piecestore: (Node ID: 12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, Piece ID: HYDMGVL463LIAO4N7JRFMFZSZ566C7DHHEMMVCPHTDUTEQYESKFQ) context deadline exceeded; piecestore: (Node ID: 1rL2DnjSgHP36Md3bCgy5padqTAhv3iQCeURbLMuuuMBxtUNZH, Piece ID: EQILGZ4H4R254DE7UW43GYCETN77D5FIRK4SE4Y36PPPSOKBSX5A) context deadline exceeded; piecestore: (Node ID: 12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n, Piece ID: RLDHK2A262X6IZQWP6DK2IPWBTMXYZ4YQQ7SG4IJRSO3KDPBUJYA) context deadline exceeded ======LOGS====== Dec 4 15:38:01.034643304 (22h42m37.505854043s) [upload=19952]: Starting upload Dec 4 15:38:01.034798069 (22h42m37.506008818s) [upload=19952]: Done starting upload Dec 4 15:38:01.034989493 (22h42m37.506200232s) [upload=19952]: Uploading segments... Dec 4 15:38:01.035032334 (22h42m37.506243073s) [upload=19952]: Got next segment. Inline: false Dec 4 15:38:01.035060508 (22h42m37.506271237s) [upload=19952]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 15:38:01.035064726 (22h42m37.506275455s) [upload=19952]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 15:38:01.301722830 (22h42m37.772933559s) [seg_pos={0 0}, upload=19952]: Begin upload segment... Dec 4 15:38:01.301843599 (22h42m37.773054338s) [seg_pos={0 0}, upload=19952]: Done begin upload segment. Dec 4 15:38:01.301883024 (22h42m37.773093753s) [noise=true , address=145.79.227.40:4051 , piece_id=RLDHK2A262X6IZQWP6DK2IPWBTMXYZ4YQQ7SG4IJRSO3KDPBUJYA, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.301945323 (22h42m37.773156092s) [noise=true , address=145.79.227.42:4121 , piece_id=USB7H5RUYH4JMFQQRO5M3BSKB4NETHDTSGV5CJRFOT76D32V6DHQ, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.302924858 (22h42m37.774135597s) [noise=true , address=145.79.227.23:4547 , piece_id=AGTUJHQKSNRQS6TR6Z6JL6HR47LYKLVQGY5WZR7Z4RBXC7NCLWYQ, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.303471950 (22h42m37.774682689s) [noise=true , address=145.79.227.34:4906 , piece_id=XNS7EVNDDK3XQVIYL5RPPIJPPTFNSTVJFWF6Q55EL3SRRAQTRYEQ, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.304017129 (22h42m37.775227868s) [noise=true , address=145.79.227.8:4330 , piece_id=3PCXR266AYJQSCQDCPBESLN7H4WLS6BLXDQWMWUOBS37SRVVKDCQ, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.304628322 (22h42m37.775839081s) [noise=true , address=77.90.31.11:4555 , piece_id=NDNU7HIDBMQGCP5HKQEGWAQCSENZQ4XPL5PDBLZSUVJNUMMKDKLA, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.305202977 (22h42m37.776413716s) [noise=true , address=145.79.227.27:4691 , piece_id=CIIRLVK6IODAMCRN7LBRLXQ5IR2AC7J3QAPHZUA3GROJIWNNEELQ, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.305742744 (22h42m37.776953483s) [noise=true , address=145.79.227.14:4840 , piece_id=7CFAXRK7F366A7QPI5OPTWMRO3Z5LK7LMEDP37AB5XLUMP34JVHQ, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.306643649 (22h42m37.777854389s) [noise=true , address=145.79.227.9:4367 , piece_id=E7UKP2XNQTELMOXMLN5CUTEG6ZVZE5TT4DW6ZYUD5733HRHWYD6Q, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.306719754 (22h42m37.777930493s) [noise=true , address=145.79.227.26:4655 , piece_id=O65JQUKSUPZSBOI5EPDDRSPMCN2LI2BYWBVDYSRIG4H6F2ASKS2A, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.307495471 (22h42m37.7787062s) [noise=true , address=145.79.227.7:4293 , piece_id=HYDMGVL463LIAO4N7JRFMFZSZ566C7DHHEMMVCPHTDUTEQYESKFQ, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.307859154 (22h42m37.779069883s) [noise=true , address=145.79.227.13:4804 , piece_id=ALPYEUHA2CMGWO6NFA7UBQUIJZTGE4NF7QYLNVAZQKYTJKABT6DA, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.308629862 (22h42m37.779840601s) [noise=true , address=145.79.227.16:4911 , piece_id=2M7N5623QL2BC6IMB3RJVRUBLB2SNZ66SWZY45N6WBYLCC56IZMA, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.310368402 (22h42m37.781579141s) [noise=true , address=145.79.227.21:4474 , piece_id=EQILGZ4H4R254DE7UW43GYCETN77D5FIRK4SE4Y36PPPSOKBSX5A, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.315008367 (22h42m37.786219106s) [noise=true , address=145.79.227.32:4834 , piece_id=CIJJAFXIJYJ6MHGVXTZ6FLU36QWE3HNVOLWW4IMUH5ZCWTL6ETZQ, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.317256889 (22h42m37.788467628s) [noise=true , address=145.79.227.31:4799 , piece_id=FBKOSDEM2H5ETZM6OOGLZCDGTT5RQSLMRZ2CLX2GOZRHRCMR2RNA, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:03.018370042 (22h42m39.489580781s) [upload=19952]: Next returned nil segment Dec 4 15:38:03.018386903 (22h42m39.489597642s) [upload=19952]: Waiting for error group managing segments... Dec 4 15:38:03.360528210 (22h42m39.831738939s) [noise=true , address=77.90.31.11:4555 , piece_id=NDNU7HIDBMQGCP5HKQEGWAQCSENZQ4XPL5PDBLZSUVJNUMMKDKLA, seg_pos={0 0}, upload=19952]: Done uploading piece. err: <nil> Dec 4 15:38:03.476269451 (22h42m39.94748018s) [noise=true , address=145.79.227.16:4911 , piece_id=2M7N5623QL2BC6IMB3RJVRUBLB2SNZ66SWZY45N6WBYLCC56IZMA, seg_pos={0 0}, upload=19952]: Done uploading piece. err: <nil> Dec 4 15:38:03.477241432 (22h42m39.948452171s) [noise=true , address=145.79.227.9:4367 , piece_id=E7UKP2XNQTELMOXMLN5CUTEG6ZVZE5TT4DW6ZYUD5733HRHWYD6Q, seg_pos={0 0}, upload=19952]: Done uploading piece. err: <nil> Dec 4 15:38:03.483785764 (22h42m39.954996503s) [noise=true , address=145.79.227.14:4840 , piece_id=7CFAXRK7F366A7QPI5OPTWMRO3Z5LK7LMEDP37AB5XLUMP34JVHQ, seg_pos={0 0}, upload=19952]: Done uploading piece. err: <nil> Dec 4 15:38:03.484258473 (22h42m39.955469212s) [noise=true , address=145.79.227.27:4691 , piece_id=CIIRLVK6IODAMCRN7LBRLXQ5IR2AC7J3QAPHZUA3GROJIWNNEELQ, seg_pos={0 0}, upload=19952]: Done uploading piece. err: <nil> Dec 4 15:38:03.487612530 (22h42m39.958823269s) [noise=true , address=145.79.227.23:4547 , piece_id=AGTUJHQKSNRQS6TR6Z6JL6HR47LYKLVQGY5WZR7Z4RBXC7NCLWYQ, seg_pos={0 0}, upload=19952]: Done uploading piece. err: <nil> Dec 4 15:38:03.495282255 (22h42m39.966492984s) [noise=true , address=145.79.227.40:4051 , piece_id=RLDHK2A262X6IZQWP6DK2IPWBTMXYZ4YQQ7SG4IJRSO3KDPBUJYA, seg_pos={0 0}, upload=19952]: Done uploading piece. err: <nil> Dec 4 15:38:03.509793507 (22h42m39.981004236s) [noise=true , address=145.79.227.8:4330 , piece_id=3PCXR266AYJQSCQDCPBESLN7H4WLS6BLXDQWMWUOBS37SRVVKDCQ, seg_pos={0 0}, upload=19952]: Done uploading piece. err: <nil> Dec 4 15:38:03.513640021 (22h42m39.98485076s) [noise=true , address=145.79.227.34:4906 , piece_id=XNS7EVNDDK3XQVIYL5RPPIJPPTFNSTVJFWF6Q55EL3SRRAQTRYEQ, seg_pos={0 0}, upload=19952]: Done uploading piece. err: <nil> Dec 4 15:38:03.785547720 (22h42m40.256758459s) [noise=true , address=145.79.227.13:4804 , piece_id=ALPYEUHA2CMGWO6NFA7UBQUIJZTGE4NF7QYLNVAZQKYTJKABT6DA, seg_pos={0 0}, upload=19952]: Done uploading piece. err: <nil> Dec 4 15:38:03.809124804 (22h42m40.280335543s) [noise=true , address=145.79.227.21:4474 , piece_id=EQILGZ4H4R254DE7UW43GYCETN77D5FIRK4SE4Y36PPPSOKBSX5A, seg_pos={0 0}, upload=19952]: Done uploading piece. err: <nil> Dec 4 15:38:04.138848766 (22h42m40.610059495s) [noise=true , address=145.79.227.7:4293 , piece_id=HYDMGVL463LIAO4N7JRFMFZSZ566C7DHHEMMVCPHTDUTEQYESKFQ, seg_pos={0 0}, upload=19952]: Done uploading piece. err: <nil> Dec 4 15:38:04.176120728 (22h42m40.647331457s) [noise=true , address=145.79.227.32:4834 , piece_id=CIJJAFXIJYJ6MHGVXTZ6FLU36QWE3HNVOLWW4IMUH5ZCWTL6ETZQ, seg_pos={0 0}, upload=19952]: Done uploading piece. err: <nil> Dec 4 15:38:04.176143252 (22h42m40.647353991s) [seg_pos={0 0}, upload=19952]: Segment reached optimal threshold of 13 pieces. Dec 4 15:38:04.176373089 (22h42m40.647583818s) [noise=true , address=145.79.227.26:4655 , piece_id=O65JQUKSUPZSBOI5EPDDRSPMCN2LI2BYWBVDYSRIG4H6F2ASKS2A, seg_pos={0 0}, upload=19952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xpZx5VbgHmbepaS9ZXgHfp7SZGeYPEjuJddQeYPUVpDU4EGx5): protocol: write tcp 188.166.225.170:46794->145.79.227.26:4655: use of closed network connection; write tcp 188.166.225.170:46794->145.79.227.26:4655: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46794->145.79.227.26:4655: use of closed network connection Dec 4 15:38:04.176382406 (22h42m40.647593145s) [noise=true , address=145.79.227.26:4655 , piece_id=O65JQUKSUPZSBOI5EPDDRSPMCN2LI2BYWBVDYSRIG4H6F2ASKS2A, seg_pos={0 0}, upload=19952]: Long tail cancellation due to optimal threshold Dec 4 15:38:04.176643925 (22h42m40.647854654s) [noise=true , address=145.79.227.31:4799 , piece_id=FBKOSDEM2H5ETZM6OOGLZCDGTT5RQSLMRZ2CLX2GOZRHRCMR2RNA, seg_pos={0 0}, upload=19952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X): protocol: expected piece hash; context canceled; EOF Dec 4 15:38:04.176651369 (22h42m40.647862108s) [noise=true , address=145.79.227.31:4799 , piece_id=FBKOSDEM2H5ETZM6OOGLZCDGTT5RQSLMRZ2CLX2GOZRHRCMR2RNA, seg_pos={0 0}, upload=19952]: Long tail cancellation due to optimal threshold Dec 4 15:38:04.176821292 (22h42m40.648032041s) [noise=true , address=145.79.227.42:4121 , piece_id=USB7H5RUYH4JMFQQRO5M3BSKB4NETHDTSGV5CJRFOT76D32V6DHQ, seg_pos={0 0}, upload=19952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh): protocol: expected piece hash; context canceled; EOF Dec 4 15:38:04.176869694 (22h42m40.648080433s) [noise=true , address=145.79.227.42:4121 , piece_id=USB7H5RUYH4JMFQQRO5M3BSKB4NETHDTSGV5CJRFOT76D32V6DHQ, seg_pos={0 0}, upload=19952]: Long tail cancellation due to optimal threshold Dec 4 15:38:04.176908829 (22h42m40.648119569s) [seg_pos={0 0}, upload=19952]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 4 15:38:04.177077430 (22h42m40.648288169s) [upload=19952]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 4 15:38:04.177087099 (22h42m40.648297848s) [upload=19952]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 4 15:38:04.502975285 (22h42m40.974186014s) [upload=19952]: Done uploading segments... Dec 4 15:38:04.502992749 (22h42m40.974203488s) [upload=19952]: Upload finished. err: <nil> Duration: 5.705450344s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 5.671982426s trace: https://jaeger.dev.storj.io/trace/1729cf59b626c72a FAIL (2025-12-04 16:08:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw, Piece ID: XNS7EVNDDK3XQVIYL5RPPIJPPTFNSTVJFWF6Q55EL3SRRAQTRYEQ) context deadline exceeded; piecestore: (Node ID: 12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, Piece ID: HYDMGVL463LIAO4N7JRFMFZSZ566C7DHHEMMVCPHTDUTEQYESKFQ) context deadline exceeded; piecestore: (Node ID: 1rL2DnjSgHP36Md3bCgy5padqTAhv3iQCeURbLMuuuMBxtUNZH, Piece ID: EQILGZ4H4R254DE7UW43GYCETN77D5FIRK4SE4Y36PPPSOKBSX5A) context deadline exceeded; piecestore: (Node ID: 12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n, Piece ID: RLDHK2A262X6IZQWP6DK2IPWBTMXYZ4YQQ7SG4IJRSO3KDPBUJYA) context deadline exceeded ======LOGS====== Dec 4 15:38:01.034643304 (22h42m37.505854043s) [upload=19952]: Starting upload Dec 4 15:38:01.034798069 (22h42m37.506008818s) [upload=19952]: Done starting upload Dec 4 15:38:01.034989493 (22h42m37.506200232s) [upload=19952]: Uploading segments... Dec 4 15:38:01.035032334 (22h42m37.506243073s) [upload=19952]: Got next segment. Inline: false Dec 4 15:38:01.035060508 (22h42m37.506271237s) [upload=19952]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 15:38:01.035064726 (22h42m37.506275455s) [upload=19952]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 15:38:01.301722830 (22h42m37.772933559s) [seg_pos={0 0}, upload=19952]: Begin upload segment... Dec 4 15:38:01.301843599 (22h42m37.773054338s) [seg_pos={0 0}, upload=19952]: Done begin upload segment. Dec 4 15:38:01.301883024 (22h42m37.773093753s) [noise=true , address=145.79.227.40:4051 , piece_id=RLDHK2A262X6IZQWP6DK2IPWBTMXYZ4YQQ7SG4IJRSO3KDPBUJYA, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.301945323 (22h42m37.773156092s) [noise=true , address=145.79.227.42:4121 , piece_id=USB7H5RUYH4JMFQQRO5M3BSKB4NETHDTSGV5CJRFOT76D32V6DHQ, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.302924858 (22h42m37.774135597s) [noise=true , address=145.79.227.23:4547 , piece_id=AGTUJHQKSNRQS6TR6Z6JL6HR47LYKLVQGY5WZR7Z4RBXC7NCLWYQ, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.303471950 (22h42m37.774682689s) [noise=true , address=145.79.227.34:4906 , piece_id=XNS7EVNDDK3XQVIYL5RPPIJPPTFNSTVJFWF6Q55EL3SRRAQTRYEQ, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.304017129 (22h42m37.775227868s) [noise=true , address=145.79.227.8:4330 , piece_id=3PCXR266AYJQSCQDCPBESLN7H4WLS6BLXDQWMWUOBS37SRVVKDCQ, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.304628322 (22h42m37.775839081s) [noise=true , address=77.90.31.11:4555 , piece_id=NDNU7HIDBMQGCP5HKQEGWAQCSENZQ4XPL5PDBLZSUVJNUMMKDKLA, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.305202977 (22h42m37.776413716s) [noise=true , address=145.79.227.27:4691 , piece_id=CIIRLVK6IODAMCRN7LBRLXQ5IR2AC7J3QAPHZUA3GROJIWNNEELQ, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.305742744 (22h42m37.776953483s) [noise=true , address=145.79.227.14:4840 , piece_id=7CFAXRK7F366A7QPI5OPTWMRO3Z5LK7LMEDP37AB5XLUMP34JVHQ, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.306643649 (22h42m37.777854389s) [noise=true , address=145.79.227.9:4367 , piece_id=E7UKP2XNQTELMOXMLN5CUTEG6ZVZE5TT4DW6ZYUD5733HRHWYD6Q, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.306719754 (22h42m37.777930493s) [noise=true , address=145.79.227.26:4655 , piece_id=O65JQUKSUPZSBOI5EPDDRSPMCN2LI2BYWBVDYSRIG4H6F2ASKS2A, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.307495471 (22h42m37.7787062s) [noise=true , address=145.79.227.7:4293 , piece_id=HYDMGVL463LIAO4N7JRFMFZSZ566C7DHHEMMVCPHTDUTEQYESKFQ, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.307859154 (22h42m37.779069883s) [noise=true , address=145.79.227.13:4804 , piece_id=ALPYEUHA2CMGWO6NFA7UBQUIJZTGE4NF7QYLNVAZQKYTJKABT6DA, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.308629862 (22h42m37.779840601s) [noise=true , address=145.79.227.16:4911 , piece_id=2M7N5623QL2BC6IMB3RJVRUBLB2SNZ66SWZY45N6WBYLCC56IZMA, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.310368402 (22h42m37.781579141s) [noise=true , address=145.79.227.21:4474 , piece_id=EQILGZ4H4R254DE7UW43GYCETN77D5FIRK4SE4Y36PPPSOKBSX5A, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.315008367 (22h42m37.786219106s) [noise=true , address=145.79.227.32:4834 , piece_id=CIJJAFXIJYJ6MHGVXTZ6FLU36QWE3HNVOLWW4IMUH5ZCWTL6ETZQ, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.317256889 (22h42m37.788467628s) [noise=true , address=145.79.227.31:4799 , piece_id=FBKOSDEM2H5ETZM6OOGLZCDGTT5RQSLMRZ2CLX2GOZRHRCMR2RNA, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:03.018370042 (22h42m39.489580781s) [upload=19952]: Next returned nil segment Dec 4 15:38:03.018386903 (22h42m39.489597642s) [upload=19952]: Waiting for error group managing segments... Dec 4 15:38:03.360528210 (22h42m39.831738939s) [noise=true , address=77.90.31.11:4555 , piece_id=NDNU7HIDBMQGCP5HKQEGWAQCSENZQ4XPL5PDBLZSUVJNUMMKDKLA, seg_pos={0 0}, upload=19952]: Done uploading piece. err: <nil> Dec 4 15:38:03.476269451 (22h42m39.94748018s) [noise=true , address=145.79.227.16:4911 , piece_id=2M7N5623QL2BC6IMB3RJVRUBLB2SNZ66SWZY45N6WBYLCC56IZMA, seg_pos={0 0}, upload=19952]: Done uploading piece. err: <nil> Dec 4 15:38:03.477241432 (22h42m39.948452171s) [noise=true , address=145.79.227.9:4367 , piece_id=E7UKP2XNQTELMOXMLN5CUTEG6ZVZE5TT4DW6ZYUD5733HRHWYD6Q, seg_pos={0 0}, upload=19952]: Done uploading piece. err: <nil> Dec 4 15:38:03.483785764 (22h42m39.954996503s) [noise=true , address=145.79.227.14:4840 , piece_id=7CFAXRK7F366A7QPI5OPTWMRO3Z5LK7LMEDP37AB5XLUMP34JVHQ, seg_pos={0 0}, upload=19952]: Done uploading piece. err: <nil> Dec 4 15:38:03.484258473 (22h42m39.955469212s) [noise=true , address=145.79.227.27:4691 , piece_id=CIIRLVK6IODAMCRN7LBRLXQ5IR2AC7J3QAPHZUA3GROJIWNNEELQ, seg_pos={0 0}, upload=19952]: Done uploading piece. err: <nil> Dec 4 15:38:03.487612530 (22h42m39.958823269s) [noise=true , address=145.79.227.23:4547 , piece_id=AGTUJHQKSNRQS6TR6Z6JL6HR47LYKLVQGY5WZR7Z4RBXC7NCLWYQ, seg_pos={0 0}, upload=19952]: Done uploading piece. err: <nil> Dec 4 15:38:03.495282255 (22h42m39.966492984s) [noise=true , address=145.79.227.40:4051 , piece_id=RLDHK2A262X6IZQWP6DK2IPWBTMXYZ4YQQ7SG4IJRSO3KDPBUJYA, seg_pos={0 0}, upload=19952]: Done uploading piece. err: <nil> Dec 4 15:38:03.509793507 (22h42m39.981004236s) [noise=true , address=145.79.227.8:4330 , piece_id=3PCXR266AYJQSCQDCPBESLN7H4WLS6BLXDQWMWUOBS37SRVVKDCQ, seg_pos={0 0}, upload=19952]: Done uploading piece. err: <nil> Dec 4 15:38:03.513640021 (22h42m39.98485076s) [noise=true , address=145.79.227.34:4906 , piece_id=XNS7EVNDDK3XQVIYL5RPPIJPPTFNSTVJFWF6Q55EL3SRRAQTRYEQ, seg_pos={0 0}, upload=19952]: Done uploading piece. err: <nil> Dec 4 15:38:03.785547720 (22h42m40.256758459s) [noise=true , address=145.79.227.13:4804 , piece_id=ALPYEUHA2CMGWO6NFA7UBQUIJZTGE4NF7QYLNVAZQKYTJKABT6DA, seg_pos={0 0}, upload=19952]: Done uploading piece. err: <nil> Dec 4 15:38:03.809124804 (22h42m40.280335543s) [noise=true , address=145.79.227.21:4474 , piece_id=EQILGZ4H4R254DE7UW43GYCETN77D5FIRK4SE4Y36PPPSOKBSX5A, seg_pos={0 0}, upload=19952]: Done uploading piece. err: <nil> Dec 4 15:38:04.138848766 (22h42m40.610059495s) [noise=true , address=145.79.227.7:4293 , piece_id=HYDMGVL463LIAO4N7JRFMFZSZ566C7DHHEMMVCPHTDUTEQYESKFQ, seg_pos={0 0}, upload=19952]: Done uploading piece. err: <nil> Dec 4 15:38:04.176120728 (22h42m40.647331457s) [noise=true , address=145.79.227.32:4834 , piece_id=CIJJAFXIJYJ6MHGVXTZ6FLU36QWE3HNVOLWW4IMUH5ZCWTL6ETZQ, seg_pos={0 0}, upload=19952]: Done uploading piece. err: <nil> Dec 4 15:38:04.176143252 (22h42m40.647353991s) [seg_pos={0 0}, upload=19952]: Segment reached optimal threshold of 13 pieces. Dec 4 15:38:04.176373089 (22h42m40.647583818s) [noise=true , address=145.79.227.26:4655 , piece_id=O65JQUKSUPZSBOI5EPDDRSPMCN2LI2BYWBVDYSRIG4H6F2ASKS2A, seg_pos={0 0}, upload=19952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xpZx5VbgHmbepaS9ZXgHfp7SZGeYPEjuJddQeYPUVpDU4EGx5): protocol: write tcp 188.166.225.170:46794->145.79.227.26:4655: use of closed network connection; write tcp 188.166.225.170:46794->145.79.227.26:4655: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46794->145.79.227.26:4655: use of closed network connection Dec 4 15:38:04.176382406 (22h42m40.647593145s) [noise=true , address=145.79.227.26:4655 , piece_id=O65JQUKSUPZSBOI5EPDDRSPMCN2LI2BYWBVDYSRIG4H6F2ASKS2A, seg_pos={0 0}, upload=19952]: Long tail cancellation due to optimal threshold Dec 4 15:38:04.176643925 (22h42m40.647854654s) [noise=true , address=145.79.227.31:4799 , piece_id=FBKOSDEM2H5ETZM6OOGLZCDGTT5RQSLMRZ2CLX2GOZRHRCMR2RNA, seg_pos={0 0}, upload=19952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X): protocol: expected piece hash; context canceled; EOF Dec 4 15:38:04.176651369 (22h42m40.647862108s) [noise=true , address=145.79.227.31:4799 , piece_id=FBKOSDEM2H5ETZM6OOGLZCDGTT5RQSLMRZ2CLX2GOZRHRCMR2RNA, seg_pos={0 0}, upload=19952]: Long tail cancellation due to optimal threshold Dec 4 15:38:04.176821292 (22h42m40.648032041s) [noise=true , address=145.79.227.42:4121 , piece_id=USB7H5RUYH4JMFQQRO5M3BSKB4NETHDTSGV5CJRFOT76D32V6DHQ, seg_pos={0 0}, upload=19952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh): protocol: expected piece hash; context canceled; EOF Dec 4 15:38:04.176869694 (22h42m40.648080433s) [noise=true , address=145.79.227.42:4121 , piece_id=USB7H5RUYH4JMFQQRO5M3BSKB4NETHDTSGV5CJRFOT76D32V6DHQ, seg_pos={0 0}, upload=19952]: Long tail cancellation due to optimal threshold Dec 4 15:38:04.176908829 (22h42m40.648119569s) [seg_pos={0 0}, upload=19952]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 4 15:38:04.177077430 (22h42m40.648288169s) [upload=19952]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 4 15:38:04.177087099 (22h42m40.648297848s) [upload=19952]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 4 15:38:04.502975285 (22h42m40.974186014s) [upload=19952]: Done uploading segments... Dec 4 15:38:04.502992749 (22h42m40.974203488s) [upload=19952]: Upload finished. err: <nil> Duration: 5.672012514s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 5.656761194s trace: https://jaeger.dev.storj.io/trace/1729cf59b626c72a FAIL (2025-12-04 16:07:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw, Piece ID: XNS7EVNDDK3XQVIYL5RPPIJPPTFNSTVJFWF6Q55EL3SRRAQTRYEQ) context deadline exceeded; piecestore: (Node ID: 12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, Piece ID: HYDMGVL463LIAO4N7JRFMFZSZ566C7DHHEMMVCPHTDUTEQYESKFQ) context deadline exceeded; piecestore: (Node ID: 1rL2DnjSgHP36Md3bCgy5padqTAhv3iQCeURbLMuuuMBxtUNZH, Piece ID: EQILGZ4H4R254DE7UW43GYCETN77D5FIRK4SE4Y36PPPSOKBSX5A) context deadline exceeded; piecestore: (Node ID: 12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n, Piece ID: RLDHK2A262X6IZQWP6DK2IPWBTMXYZ4YQQ7SG4IJRSO3KDPBUJYA) context deadline exceeded ======LOGS====== Dec 4 15:38:01.034643304 (22h42m37.505854043s) [upload=19952]: Starting upload Dec 4 15:38:01.034798069 (22h42m37.506008818s) [upload=19952]: Done starting upload Dec 4 15:38:01.034989493 (22h42m37.506200232s) [upload=19952]: Uploading segments... Dec 4 15:38:01.035032334 (22h42m37.506243073s) [upload=19952]: Got next segment. Inline: false Dec 4 15:38:01.035060508 (22h42m37.506271237s) [upload=19952]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 15:38:01.035064726 (22h42m37.506275455s) [upload=19952]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 15:38:01.301722830 (22h42m37.772933559s) [seg_pos={0 0}, upload=19952]: Begin upload segment... Dec 4 15:38:01.301843599 (22h42m37.773054338s) [seg_pos={0 0}, upload=19952]: Done begin upload segment. Dec 4 15:38:01.301883024 (22h42m37.773093753s) [noise=true , address=145.79.227.40:4051 , piece_id=RLDHK2A262X6IZQWP6DK2IPWBTMXYZ4YQQ7SG4IJRSO3KDPBUJYA, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.301945323 (22h42m37.773156092s) [noise=true , address=145.79.227.42:4121 , piece_id=USB7H5RUYH4JMFQQRO5M3BSKB4NETHDTSGV5CJRFOT76D32V6DHQ, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.302924858 (22h42m37.774135597s) [noise=true , address=145.79.227.23:4547 , piece_id=AGTUJHQKSNRQS6TR6Z6JL6HR47LYKLVQGY5WZR7Z4RBXC7NCLWYQ, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.303471950 (22h42m37.774682689s) [noise=true , address=145.79.227.34:4906 , piece_id=XNS7EVNDDK3XQVIYL5RPPIJPPTFNSTVJFWF6Q55EL3SRRAQTRYEQ, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.304017129 (22h42m37.775227868s) [noise=true , address=145.79.227.8:4330 , piece_id=3PCXR266AYJQSCQDCPBESLN7H4WLS6BLXDQWMWUOBS37SRVVKDCQ, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.304628322 (22h42m37.775839081s) [noise=true , address=77.90.31.11:4555 , piece_id=NDNU7HIDBMQGCP5HKQEGWAQCSENZQ4XPL5PDBLZSUVJNUMMKDKLA, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.305202977 (22h42m37.776413716s) [noise=true , address=145.79.227.27:4691 , piece_id=CIIRLVK6IODAMCRN7LBRLXQ5IR2AC7J3QAPHZUA3GROJIWNNEELQ, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.305742744 (22h42m37.776953483s) [noise=true , address=145.79.227.14:4840 , piece_id=7CFAXRK7F366A7QPI5OPTWMRO3Z5LK7LMEDP37AB5XLUMP34JVHQ, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.306643649 (22h42m37.777854389s) [noise=true , address=145.79.227.9:4367 , piece_id=E7UKP2XNQTELMOXMLN5CUTEG6ZVZE5TT4DW6ZYUD5733HRHWYD6Q, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.306719754 (22h42m37.777930493s) [noise=true , address=145.79.227.26:4655 , piece_id=O65JQUKSUPZSBOI5EPDDRSPMCN2LI2BYWBVDYSRIG4H6F2ASKS2A, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.307495471 (22h42m37.7787062s) [noise=true , address=145.79.227.7:4293 , piece_id=HYDMGVL463LIAO4N7JRFMFZSZ566C7DHHEMMVCPHTDUTEQYESKFQ, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.307859154 (22h42m37.779069883s) [noise=true , address=145.79.227.13:4804 , piece_id=ALPYEUHA2CMGWO6NFA7UBQUIJZTGE4NF7QYLNVAZQKYTJKABT6DA, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.308629862 (22h42m37.779840601s) [noise=true , address=145.79.227.16:4911 , piece_id=2M7N5623QL2BC6IMB3RJVRUBLB2SNZ66SWZY45N6WBYLCC56IZMA, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.310368402 (22h42m37.781579141s) [noise=true , address=145.79.227.21:4474 , piece_id=EQILGZ4H4R254DE7UW43GYCETN77D5FIRK4SE4Y36PPPSOKBSX5A, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.315008367 (22h42m37.786219106s) [noise=true , address=145.79.227.32:4834 , piece_id=CIJJAFXIJYJ6MHGVXTZ6FLU36QWE3HNVOLWW4IMUH5ZCWTL6ETZQ, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.317256889 (22h42m37.788467628s) [noise=true , address=145.79.227.31:4799 , piece_id=FBKOSDEM2H5ETZM6OOGLZCDGTT5RQSLMRZ2CLX2GOZRHRCMR2RNA, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:03.018370042 (22h42m39.489580781s) [upload=19952]: Next returned nil segment Dec 4 15:38:03.018386903 (22h42m39.489597642s) [upload=19952]: Waiting for error group managing segments... Dec 4 15:38:03.360528210 (22h42m39.831738939s) [noise=true , address=77.90.31.11:4555 , piece_id=NDNU7HIDBMQGCP5HKQEGWAQCSENZQ4XPL5PDBLZSUVJNUMMKDKLA, seg_pos={0 0}, upload=19952]: Done uploading piece. err: <nil> Dec 4 15:38:03.476269451 (22h42m39.94748018s) [noise=true , address=145.79.227.16:4911 , piece_id=2M7N5623QL2BC6IMB3RJVRUBLB2SNZ66SWZY45N6WBYLCC56IZMA, seg_pos={0 0}, upload=19952]: Done uploading piece. err: <nil> Dec 4 15:38:03.477241432 (22h42m39.948452171s) [noise=true , address=145.79.227.9:4367 , piece_id=E7UKP2XNQTELMOXMLN5CUTEG6ZVZE5TT4DW6ZYUD5733HRHWYD6Q, seg_pos={0 0}, upload=19952]: Done uploading piece. err: <nil> Dec 4 15:38:03.483785764 (22h42m39.954996503s) [noise=true , address=145.79.227.14:4840 , piece_id=7CFAXRK7F366A7QPI5OPTWMRO3Z5LK7LMEDP37AB5XLUMP34JVHQ, seg_pos={0 0}, upload=19952]: Done uploading piece. err: <nil> Dec 4 15:38:03.484258473 (22h42m39.955469212s) [noise=true , address=145.79.227.27:4691 , piece_id=CIIRLVK6IODAMCRN7LBRLXQ5IR2AC7J3QAPHZUA3GROJIWNNEELQ, seg_pos={0 0}, upload=19952]: Done uploading piece. err: <nil> Dec 4 15:38:03.487612530 (22h42m39.958823269s) [noise=true , address=145.79.227.23:4547 , piece_id=AGTUJHQKSNRQS6TR6Z6JL6HR47LYKLVQGY5WZR7Z4RBXC7NCLWYQ, seg_pos={0 0}, upload=19952]: Done uploading piece. err: <nil> Dec 4 15:38:03.495282255 (22h42m39.966492984s) [noise=true , address=145.79.227.40:4051 , piece_id=RLDHK2A262X6IZQWP6DK2IPWBTMXYZ4YQQ7SG4IJRSO3KDPBUJYA, seg_pos={0 0}, upload=19952]: Done uploading piece. err: <nil> Dec 4 15:38:03.509793507 (22h42m39.981004236s) [noise=true , address=145.79.227.8:4330 , piece_id=3PCXR266AYJQSCQDCPBESLN7H4WLS6BLXDQWMWUOBS37SRVVKDCQ, seg_pos={0 0}, upload=19952]: Done uploading piece. err: <nil> Dec 4 15:38:03.513640021 (22h42m39.98485076s) [noise=true , address=145.79.227.34:4906 , piece_id=XNS7EVNDDK3XQVIYL5RPPIJPPTFNSTVJFWF6Q55EL3SRRAQTRYEQ, seg_pos={0 0}, upload=19952]: Done uploading piece. err: <nil> Dec 4 15:38:03.785547720 (22h42m40.256758459s) [noise=true , address=145.79.227.13:4804 , piece_id=ALPYEUHA2CMGWO6NFA7UBQUIJZTGE4NF7QYLNVAZQKYTJKABT6DA, seg_pos={0 0}, upload=19952]: Done uploading piece. err: <nil> Dec 4 15:38:03.809124804 (22h42m40.280335543s) [noise=true , address=145.79.227.21:4474 , piece_id=EQILGZ4H4R254DE7UW43GYCETN77D5FIRK4SE4Y36PPPSOKBSX5A, seg_pos={0 0}, upload=19952]: Done uploading piece. err: <nil> Dec 4 15:38:04.138848766 (22h42m40.610059495s) [noise=true , address=145.79.227.7:4293 , piece_id=HYDMGVL463LIAO4N7JRFMFZSZ566C7DHHEMMVCPHTDUTEQYESKFQ, seg_pos={0 0}, upload=19952]: Done uploading piece. err: <nil> Dec 4 15:38:04.176120728 (22h42m40.647331457s) [noise=true , address=145.79.227.32:4834 , piece_id=CIJJAFXIJYJ6MHGVXTZ6FLU36QWE3HNVOLWW4IMUH5ZCWTL6ETZQ, seg_pos={0 0}, upload=19952]: Done uploading piece. err: <nil> Dec 4 15:38:04.176143252 (22h42m40.647353991s) [seg_pos={0 0}, upload=19952]: Segment reached optimal threshold of 13 pieces. Dec 4 15:38:04.176373089 (22h42m40.647583818s) [noise=true , address=145.79.227.26:4655 , piece_id=O65JQUKSUPZSBOI5EPDDRSPMCN2LI2BYWBVDYSRIG4H6F2ASKS2A, seg_pos={0 0}, upload=19952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xpZx5VbgHmbepaS9ZXgHfp7SZGeYPEjuJddQeYPUVpDU4EGx5): protocol: write tcp 188.166.225.170:46794->145.79.227.26:4655: use of closed network connection; write tcp 188.166.225.170:46794->145.79.227.26:4655: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46794->145.79.227.26:4655: use of closed network connection Dec 4 15:38:04.176382406 (22h42m40.647593145s) [noise=true , address=145.79.227.26:4655 , piece_id=O65JQUKSUPZSBOI5EPDDRSPMCN2LI2BYWBVDYSRIG4H6F2ASKS2A, seg_pos={0 0}, upload=19952]: Long tail cancellation due to optimal threshold Dec 4 15:38:04.176643925 (22h42m40.647854654s) [noise=true , address=145.79.227.31:4799 , piece_id=FBKOSDEM2H5ETZM6OOGLZCDGTT5RQSLMRZ2CLX2GOZRHRCMR2RNA, seg_pos={0 0}, upload=19952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X): protocol: expected piece hash; context canceled; EOF Dec 4 15:38:04.176651369 (22h42m40.647862108s) [noise=true , address=145.79.227.31:4799 , piece_id=FBKOSDEM2H5ETZM6OOGLZCDGTT5RQSLMRZ2CLX2GOZRHRCMR2RNA, seg_pos={0 0}, upload=19952]: Long tail cancellation due to optimal threshold Dec 4 15:38:04.176821292 (22h42m40.648032041s) [noise=true , address=145.79.227.42:4121 , piece_id=USB7H5RUYH4JMFQQRO5M3BSKB4NETHDTSGV5CJRFOT76D32V6DHQ, seg_pos={0 0}, upload=19952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh): protocol: expected piece hash; context canceled; EOF Dec 4 15:38:04.176869694 (22h42m40.648080433s) [noise=true , address=145.79.227.42:4121 , piece_id=USB7H5RUYH4JMFQQRO5M3BSKB4NETHDTSGV5CJRFOT76D32V6DHQ, seg_pos={0 0}, upload=19952]: Long tail cancellation due to optimal threshold Dec 4 15:38:04.176908829 (22h42m40.648119569s) [seg_pos={0 0}, upload=19952]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 4 15:38:04.177077430 (22h42m40.648288169s) [upload=19952]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 4 15:38:04.177087099 (22h42m40.648297848s) [upload=19952]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 4 15:38:04.502975285 (22h42m40.974186014s) [upload=19952]: Done uploading segments... Dec 4 15:38:04.502992749 (22h42m40.974203488s) [upload=19952]: Upload finished. err: <nil> Duration: 5.656788756s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 5.786067566s trace: https://jaeger.dev.storj.io/trace/1729cf59b626c72a FAIL (2025-12-04 16:06:58 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw, Piece ID: XNS7EVNDDK3XQVIYL5RPPIJPPTFNSTVJFWF6Q55EL3SRRAQTRYEQ) context deadline exceeded; piecestore: (Node ID: 12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, Piece ID: HYDMGVL463LIAO4N7JRFMFZSZ566C7DHHEMMVCPHTDUTEQYESKFQ) context deadline exceeded; piecestore: (Node ID: 1rL2DnjSgHP36Md3bCgy5padqTAhv3iQCeURbLMuuuMBxtUNZH, Piece ID: EQILGZ4H4R254DE7UW43GYCETN77D5FIRK4SE4Y36PPPSOKBSX5A) context deadline exceeded; piecestore: (Node ID: 12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n, Piece ID: RLDHK2A262X6IZQWP6DK2IPWBTMXYZ4YQQ7SG4IJRSO3KDPBUJYA) context deadline exceeded ======LOGS====== Dec 4 15:38:01.034643304 (22h42m37.505854043s) [upload=19952]: Starting upload Dec 4 15:38:01.034798069 (22h42m37.506008818s) [upload=19952]: Done starting upload Dec 4 15:38:01.034989493 (22h42m37.506200232s) [upload=19952]: Uploading segments... Dec 4 15:38:01.035032334 (22h42m37.506243073s) [upload=19952]: Got next segment. Inline: false Dec 4 15:38:01.035060508 (22h42m37.506271237s) [upload=19952]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 4 15:38:01.035064726 (22h42m37.506275455s) [upload=19952]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 4 15:38:01.301722830 (22h42m37.772933559s) [seg_pos={0 0}, upload=19952]: Begin upload segment... Dec 4 15:38:01.301843599 (22h42m37.773054338s) [seg_pos={0 0}, upload=19952]: Done begin upload segment. Dec 4 15:38:01.301883024 (22h42m37.773093753s) [noise=true , address=145.79.227.40:4051 , piece_id=RLDHK2A262X6IZQWP6DK2IPWBTMXYZ4YQQ7SG4IJRSO3KDPBUJYA, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.301945323 (22h42m37.773156092s) [noise=true , address=145.79.227.42:4121 , piece_id=USB7H5RUYH4JMFQQRO5M3BSKB4NETHDTSGV5CJRFOT76D32V6DHQ, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.302924858 (22h42m37.774135597s) [noise=true , address=145.79.227.23:4547 , piece_id=AGTUJHQKSNRQS6TR6Z6JL6HR47LYKLVQGY5WZR7Z4RBXC7NCLWYQ, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.303471950 (22h42m37.774682689s) [noise=true , address=145.79.227.34:4906 , piece_id=XNS7EVNDDK3XQVIYL5RPPIJPPTFNSTVJFWF6Q55EL3SRRAQTRYEQ, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.304017129 (22h42m37.775227868s) [noise=true , address=145.79.227.8:4330 , piece_id=3PCXR266AYJQSCQDCPBESLN7H4WLS6BLXDQWMWUOBS37SRVVKDCQ, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.304628322 (22h42m37.775839081s) [noise=true , address=77.90.31.11:4555 , piece_id=NDNU7HIDBMQGCP5HKQEGWAQCSENZQ4XPL5PDBLZSUVJNUMMKDKLA, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.305202977 (22h42m37.776413716s) [noise=true , address=145.79.227.27:4691 , piece_id=CIIRLVK6IODAMCRN7LBRLXQ5IR2AC7J3QAPHZUA3GROJIWNNEELQ, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.305742744 (22h42m37.776953483s) [noise=true , address=145.79.227.14:4840 , piece_id=7CFAXRK7F366A7QPI5OPTWMRO3Z5LK7LMEDP37AB5XLUMP34JVHQ, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.306643649 (22h42m37.777854389s) [noise=true , address=145.79.227.9:4367 , piece_id=E7UKP2XNQTELMOXMLN5CUTEG6ZVZE5TT4DW6ZYUD5733HRHWYD6Q, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.306719754 (22h42m37.777930493s) [noise=true , address=145.79.227.26:4655 , piece_id=O65JQUKSUPZSBOI5EPDDRSPMCN2LI2BYWBVDYSRIG4H6F2ASKS2A, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.307495471 (22h42m37.7787062s) [noise=true , address=145.79.227.7:4293 , piece_id=HYDMGVL463LIAO4N7JRFMFZSZ566C7DHHEMMVCPHTDUTEQYESKFQ, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.307859154 (22h42m37.779069883s) [noise=true , address=145.79.227.13:4804 , piece_id=ALPYEUHA2CMGWO6NFA7UBQUIJZTGE4NF7QYLNVAZQKYTJKABT6DA, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.308629862 (22h42m37.779840601s) [noise=true , address=145.79.227.16:4911 , piece_id=2M7N5623QL2BC6IMB3RJVRUBLB2SNZ66SWZY45N6WBYLCC56IZMA, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.310368402 (22h42m37.781579141s) [noise=true , address=145.79.227.21:4474 , piece_id=EQILGZ4H4R254DE7UW43GYCETN77D5FIRK4SE4Y36PPPSOKBSX5A, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.315008367 (22h42m37.786219106s) [noise=true , address=145.79.227.32:4834 , piece_id=CIJJAFXIJYJ6MHGVXTZ6FLU36QWE3HNVOLWW4IMUH5ZCWTL6ETZQ, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:01.317256889 (22h42m37.788467628s) [noise=true , address=145.79.227.31:4799 , piece_id=FBKOSDEM2H5ETZM6OOGLZCDGTT5RQSLMRZ2CLX2GOZRHRCMR2RNA, seg_pos={0 0}, upload=19952]: Uploading piece... Dec 4 15:38:03.018370042 (22h42m39.489580781s) [upload=19952]: Next returned nil segment Dec 4 15:38:03.018386903 (22h42m39.489597642s) [upload=19952]: Waiting for error group managing segments... Dec 4 15:38:03.360528210 (22h42m39.831738939s) [noise=true , address=77.90.31.11:4555 , piece_id=NDNU7HIDBMQGCP5HKQEGWAQCSENZQ4XPL5PDBLZSUVJNUMMKDKLA, seg_pos={0 0}, upload=19952]: Done uploading piece. err: <nil> Dec 4 15:38:03.476269451 (22h42m39.94748018s) [noise=true , address=145.79.227.16:4911 , piece_id=2M7N5623QL2BC6IMB3RJVRUBLB2SNZ66SWZY45N6WBYLCC56IZMA, seg_pos={0 0}, upload=19952]: Done uploading piece. err: <nil> Dec 4 15:38:03.477241432 (22h42m39.948452171s) [noise=true , address=145.79.227.9:4367 , piece_id=E7UKP2XNQTELMOXMLN5CUTEG6ZVZE5TT4DW6ZYUD5733HRHWYD6Q, seg_pos={0 0}, upload=19952]: Done uploading piece. err: <nil> Dec 4 15:38:03.483785764 (22h42m39.954996503s) [noise=true , address=145.79.227.14:4840 , piece_id=7CFAXRK7F366A7QPI5OPTWMRO3Z5LK7LMEDP37AB5XLUMP34JVHQ, seg_pos={0 0}, upload=19952]: Done uploading piece. err: <nil> Dec 4 15:38:03.484258473 (22h42m39.955469212s) [noise=true , address=145.79.227.27:4691 , piece_id=CIIRLVK6IODAMCRN7LBRLXQ5IR2AC7J3QAPHZUA3GROJIWNNEELQ, seg_pos={0 0}, upload=19952]: Done uploading piece. err: <nil> Dec 4 15:38:03.487612530 (22h42m39.958823269s) [noise=true , address=145.79.227.23:4547 , piece_id=AGTUJHQKSNRQS6TR6Z6JL6HR47LYKLVQGY5WZR7Z4RBXC7NCLWYQ, seg_pos={0 0}, upload=19952]: Done uploading piece. err: <nil> Dec 4 15:38:03.495282255 (22h42m39.966492984s) [noise=true , address=145.79.227.40:4051 , piece_id=RLDHK2A262X6IZQWP6DK2IPWBTMXYZ4YQQ7SG4IJRSO3KDPBUJYA, seg_pos={0 0}, upload=19952]: Done uploading piece. err: <nil> Dec 4 15:38:03.509793507 (22h42m39.981004236s) [noise=true , address=145.79.227.8:4330 , piece_id=3PCXR266AYJQSCQDCPBESLN7H4WLS6BLXDQWMWUOBS37SRVVKDCQ, seg_pos={0 0}, upload=19952]: Done uploading piece. err: <nil> Dec 4 15:38:03.513640021 (22h42m39.98485076s) [noise=true , address=145.79.227.34:4906 , piece_id=XNS7EVNDDK3XQVIYL5RPPIJPPTFNSTVJFWF6Q55EL3SRRAQTRYEQ, seg_pos={0 0}, upload=19952]: Done uploading piece. err: <nil> Dec 4 15:38:03.785547720 (22h42m40.256758459s) [noise=true , address=145.79.227.13:4804 , piece_id=ALPYEUHA2CMGWO6NFA7UBQUIJZTGE4NF7QYLNVAZQKYTJKABT6DA, seg_pos={0 0}, upload=19952]: Done uploading piece. err: <nil> Dec 4 15:38:03.809124804 (22h42m40.280335543s) [noise=true , address=145.79.227.21:4474 , piece_id=EQILGZ4H4R254DE7UW43GYCETN77D5FIRK4SE4Y36PPPSOKBSX5A, seg_pos={0 0}, upload=19952]: Done uploading piece. err: <nil> Dec 4 15:38:04.138848766 (22h42m40.610059495s) [noise=true , address=145.79.227.7:4293 , piece_id=HYDMGVL463LIAO4N7JRFMFZSZ566C7DHHEMMVCPHTDUTEQYESKFQ, seg_pos={0 0}, upload=19952]: Done uploading piece. err: <nil> Dec 4 15:38:04.176120728 (22h42m40.647331457s) [noise=true , address=145.79.227.32:4834 , piece_id=CIJJAFXIJYJ6MHGVXTZ6FLU36QWE3HNVOLWW4IMUH5ZCWTL6ETZQ, seg_pos={0 0}, upload=19952]: Done uploading piece. err: <nil> Dec 4 15:38:04.176143252 (22h42m40.647353991s) [seg_pos={0 0}, upload=19952]: Segment reached optimal threshold of 13 pieces. Dec 4 15:38:04.176373089 (22h42m40.647583818s) [noise=true , address=145.79.227.26:4655 , piece_id=O65JQUKSUPZSBOI5EPDDRSPMCN2LI2BYWBVDYSRIG4H6F2ASKS2A, seg_pos={0 0}, upload=19952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xpZx5VbgHmbepaS9ZXgHfp7SZGeYPEjuJddQeYPUVpDU4EGx5): protocol: write tcp 188.166.225.170:46794->145.79.227.26:4655: use of closed network connection; write tcp 188.166.225.170:46794->145.79.227.26:4655: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46794->145.79.227.26:4655: use of closed network connection Dec 4 15:38:04.176382406 (22h42m40.647593145s) [noise=true , address=145.79.227.26:4655 , piece_id=O65JQUKSUPZSBOI5EPDDRSPMCN2LI2BYWBVDYSRIG4H6F2ASKS2A, seg_pos={0 0}, upload=19952]: Long tail cancellation due to optimal threshold Dec 4 15:38:04.176643925 (22h42m40.647854654s) [noise=true , address=145.79.227.31:4799 , piece_id=FBKOSDEM2H5ETZM6OOGLZCDGTT5RQSLMRZ2CLX2GOZRHRCMR2RNA, seg_pos={0 0}, upload=19952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X): protocol: expected piece hash; context canceled; EOF Dec 4 15:38:04.176651369 (22h42m40.647862108s) [noise=true , address=145.79.227.31:4799 , piece_id=FBKOSDEM2H5ETZM6OOGLZCDGTT5RQSLMRZ2CLX2GOZRHRCMR2RNA, seg_pos={0 0}, upload=19952]: Long tail cancellation due to optimal threshold Dec 4 15:38:04.176821292 (22h42m40.648032041s) [noise=true , address=145.79.227.42:4121 , piece_id=USB7H5RUYH4JMFQQRO5M3BSKB4NETHDTSGV5CJRFOT76D32V6DHQ, seg_pos={0 0}, upload=19952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh): protocol: expected piece hash; context canceled; EOF Dec 4 15:38:04.176869694 (22h42m40.648080433s) [noise=true , address=145.79.227.42:4121 , piece_id=USB7H5RUYH4JMFQQRO5M3BSKB4NETHDTSGV5CJRFOT76D32V6DHQ, seg_pos={0 0}, upload=19952]: Long tail cancellation due to optimal threshold Dec 4 15:38:04.176908829 (22h42m40.648119569s) [seg_pos={0 0}, upload=19952]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 4 15:38:04.177077430 (22h42m40.648288169s) [upload=19952]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 4 15:38:04.177087099 (22h42m40.648297848s) [upload=19952]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 4 15:38:04.502975285 (22h42m40.974186014s) [upload=19952]: Done uploading segments... Dec 4 15:38:04.502992749 (22h42m40.974203488s) [upload=19952]: Upload finished. err: <nil> Duration: 5.786098305s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)