Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
24 out of 60 is failed (threshold=3) [X.X.......X....X..X..X....X.XXXXX.X.....X.X...XXXX..XXXX..X.] duration:
6.148723026s trace:
https://jaeger.dev.storj.io/trace/1c0826e28e8f88b4
FAIL (2025-12-31 03:58:37 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: ZTILSDDYFPH2WW64Y4636D5A3C6J4ULRRMZMZXFDCJ745A2FA7GA) context deadline exceeded; piecestore: (Node ID: 1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, Piece ID: CX7T7GQAZKFJDAYYRJ5CEXHPYQQMRMVEKBJRPTXI54H4DN4TMIYA) context deadline exceeded
======LOGS======
Dec 31 03:58:00.294001490 (612h29m0.165084525s) [upload=536743]: Starting upload
Dec 31 03:58:00.294080951 (612h29m0.165163996s) [upload=536743]: Done starting upload
Dec 31 03:58:00.294225086 (612h29m0.165308131s) [upload=536743]: Uploading segments...
Dec 31 03:58:00.294383307 (612h29m0.165466352s) [upload=536743]: Got next segment. Inline: false
Dec 31 03:58:00.294598156 (612h29m0.165681211s) [upload=536743]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 31 03:58:00.294606442 (612h29m0.165689487s) [upload=536743]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 31 03:58:00.542690384 (612h29m0.413773429s) [seg_pos={0 0}, upload=536743]: Begin upload segment...
Dec 31 03:58:00.542823247 (612h29m0.413906292s) [seg_pos={0 0}, upload=536743]: Done begin upload segment.
Dec 31 03:58:00.542914311 (612h29m0.413997356s) [noise=true , address=45.91.116.11:4119 , piece_id=FN44SD7CX4OS2YCOBQKT57O3XV4E3LBUIGX4D6FQHQDSSZGWYKPA, seg_pos={0 0}, upload=536743]: Uploading piece...
Dec 31 03:58:00.552138895 (612h29m0.42322194s) [noise=true , address=145.79.227.19:4077 , piece_id=D6QSNOT6P6643RMYKLT5ENAWCARLE37LQX4G6W23CTLZ3T4CTS5A, seg_pos={0 0}, upload=536743]: Uploading piece...
Dec 31 03:58:00.554471978 (612h29m0.425555013s) [noise=true , address=145.79.227.37:4566 , piece_id=O6PN2OLLMOISZ6XAJAEGG6XZ3M6646AAMU3L7X7KDC6MBSDLUGQA, seg_pos={0 0}, upload=536743]: Uploading piece...
Dec 31 03:58:00.554697908 (612h29m0.425780953s) [noise=true , address=45.91.116.6:4975 , piece_id=ZTILSDDYFPH2WW64Y4636D5A3C6J4ULRRMZMZXFDCJ745A2FA7GA, seg_pos={0 0}, upload=536743]: Uploading piece...
Dec 31 03:58:00.559476197 (612h29m0.430559242s) [noise=true , address=45.91.116.7:4009 , piece_id=CX7T7GQAZKFJDAYYRJ5CEXHPYQQMRMVEKBJRPTXI54H4DN4TMIYA, seg_pos={0 0}, upload=536743]: Uploading piece...
Dec 31 03:58:00.569351681 (612h29m0.440434726s) [noise=true , address=145.79.227.33:4869 , piece_id=5E6ZGJJXNRPHBMXZY6T3R3NOSMYFK7MQAEJK75KIMQURI2WP5JYQ, seg_pos={0 0}, upload=536743]: Uploading piece...
Dec 31 03:58:00.574259907 (612h29m0.445342952s) [noise=true , address=145.79.227.14:4839 , piece_id=XRHMRIBUKIJ4RG7LIMVDPPHGDWTC55TSEDDBKKMD6DJXIHNQ7RMQ, seg_pos={0 0}, upload=536743]: Uploading piece...
Dec 31 03:58:00.582068806 (612h29m0.453151851s) [noise=true , address=45.91.116.10:4083 , piece_id=SMTUKOKE6UJBFAWUFODMBYYTWNIBBXYAIYYCITBFKLH6LXXTMS3A, seg_pos={0 0}, upload=536743]: Uploading piece...
Dec 31 03:58:00.591056769 (612h29m0.462139845s) [noise=true , address=145.79.227.50:4373 , piece_id=Q4H7UMU2EEPUEAL57ITHQXRAFRK5BZXMNH6U7BGURPJAN4NCYMLA, seg_pos={0 0}, upload=536743]: Uploading piece...
Dec 31 03:58:00.601141192 (612h29m0.472224257s) [noise=true , address=45.91.116.12:4228 , piece_id=A2JARVBXJ4M3UCJLHAQIRBAHVY6L7OFAFI2LVDCZIXRM65QIWAVA, seg_pos={0 0}, upload=536743]: Uploading piece...
Dec 31 03:58:00.603482120 (612h29m0.474565165s) [noise=true , address=145.79.227.16:4909 , piece_id=R25YR4ZN6DQN7P3MQ5K6AOYOGYZDLV7TXM43NVHNCQSEQTYD6BSA, seg_pos={0 0}, upload=536743]: Uploading piece...
Dec 31 03:58:00.604129722 (612h29m0.475212757s) [noise=true , address=77.90.31.10:4518 , piece_id=C6CYAKMHVAYHK7OYFA672G4DBJXHAZ27SAFHXMZOBXW7CHSCI4QA, seg_pos={0 0}, upload=536743]: Uploading piece...
Dec 31 03:58:00.605368201 (612h29m0.476451246s) [noise=true , address=145.79.227.41:4085 , piece_id=XT3EU7WIT7FDTUC5J3Y5WI7CCSGH6ZAMU3BO5UJUDM2WYFOMLIVA, seg_pos={0 0}, upload=536743]: Uploading piece...
Dec 31 03:58:00.605539427 (612h29m0.476622462s) [noise=true , address=45.91.116.14:4191 , piece_id=WUDOWLIQEKSZY74HO7MQSRT7JE2LAWVFSZEA7VBT2M745VA4QJCA, seg_pos={0 0}, upload=536743]: Uploading piece...
Dec 31 03:58:00.607234183 (612h29m0.478317228s) [noise=true , address=145.79.227.15:4875 , piece_id=XTZBZPOCI7RK3O6LMPYLICJ4NY7WN7RNRHBWBZ7KKWIYFN2U77WA, seg_pos={0 0}, upload=536743]: Uploading piece...
Dec 31 03:58:00.608998473 (612h29m0.480081518s) [noise=true , address=145.79.227.31:4799 , piece_id=57TLZF7KTIETFEKNPW2XE4KATHNMJWF7ZB35I7CL3ZAANBOWYRSA, seg_pos={0 0}, upload=536743]: Uploading piece...
Dec 31 03:58:02.217068958 (612h29m2.088152004s) [upload=536743]: Next returned nil segment
Dec 31 03:58:02.217082103 (612h29m2.088165148s) [upload=536743]: Waiting for error group managing segments...
Dec 31 03:58:02.594337594 (612h29m2.465420639s) [noise=true , address=145.79.227.37:4566 , piece_id=O6PN2OLLMOISZ6XAJAEGG6XZ3M6646AAMU3L7X7KDC6MBSDLUGQA, seg_pos={0 0}, upload=536743]: Done uploading piece. err: <nil>
Dec 31 03:58:02.605087353 (612h29m2.476170398s) [noise=true , address=145.79.227.19:4077 , piece_id=D6QSNOT6P6643RMYKLT5ENAWCARLE37LQX4G6W23CTLZ3T4CTS5A, seg_pos={0 0}, upload=536743]: Done uploading piece. err: <nil>
Dec 31 03:58:02.607303622 (612h29m2.478386668s) [noise=true , address=145.79.227.16:4909 , piece_id=R25YR4ZN6DQN7P3MQ5K6AOYOGYZDLV7TXM43NVHNCQSEQTYD6BSA, seg_pos={0 0}, upload=536743]: Done uploading piece. err: <nil>
Dec 31 03:58:02.616067511 (612h29m2.487150546s) [noise=true , address=145.79.227.50:4373 , piece_id=Q4H7UMU2EEPUEAL57ITHQXRAFRK5BZXMNH6U7BGURPJAN4NCYMLA, seg_pos={0 0}, upload=536743]: Done uploading piece. err: <nil>
Dec 31 03:58:02.624055831 (612h29m2.495138876s) [noise=true , address=145.79.227.31:4799 , piece_id=57TLZF7KTIETFEKNPW2XE4KATHNMJWF7ZB35I7CL3ZAANBOWYRSA, seg_pos={0 0}, upload=536743]: Done uploading piece. err: <nil>
Dec 31 03:58:02.626248126 (612h29m2.497331171s) [noise=true , address=145.79.227.41:4085 , piece_id=XT3EU7WIT7FDTUC5J3Y5WI7CCSGH6ZAMU3BO5UJUDM2WYFOMLIVA, seg_pos={0 0}, upload=536743]: Done uploading piece. err: <nil>
Dec 31 03:58:02.673126778 (612h29m2.544209823s) [noise=true , address=145.79.227.14:4839 , piece_id=XRHMRIBUKIJ4RG7LIMVDPPHGDWTC55TSEDDBKKMD6DJXIHNQ7RMQ, seg_pos={0 0}, upload=536743]: Done uploading piece. err: <nil>
Dec 31 03:58:02.681227793 (612h29m2.552310838s) [noise=true , address=145.79.227.15:4875 , piece_id=XTZBZPOCI7RK3O6LMPYLICJ4NY7WN7RNRHBWBZ7KKWIYFN2U77WA, seg_pos={0 0}, upload=536743]: Done uploading piece. err: <nil>
Dec 31 03:58:02.902823439 (612h29m2.773906484s) [noise=true , address=45.91.116.6:4975 , piece_id=ZTILSDDYFPH2WW64Y4636D5A3C6J4ULRRMZMZXFDCJ745A2FA7GA, seg_pos={0 0}, upload=536743]: Done uploading piece. err: <nil>
Dec 31 03:58:02.924549789 (612h29m2.795632844s) [noise=true , address=45.91.116.12:4228 , piece_id=A2JARVBXJ4M3UCJLHAQIRBAHVY6L7OFAFI2LVDCZIXRM65QIWAVA, seg_pos={0 0}, upload=536743]: Done uploading piece. err: <nil>
Dec 31 03:58:02.937932651 (612h29m2.809015696s) [noise=true , address=45.91.116.10:4083 , piece_id=SMTUKOKE6UJBFAWUFODMBYYTWNIBBXYAIYYCITBFKLH6LXXTMS3A, seg_pos={0 0}, upload=536743]: Done uploading piece. err: <nil>
Dec 31 03:58:02.940111910 (612h29m2.811194956s) [noise=true , address=45.91.116.7:4009 , piece_id=CX7T7GQAZKFJDAYYRJ5CEXHPYQQMRMVEKBJRPTXI54H4DN4TMIYA, seg_pos={0 0}, upload=536743]: Done uploading piece. err: <nil>
Dec 31 03:58:03.091807317 (612h29m2.962890362s) [noise=true , address=45.91.116.11:4119 , piece_id=FN44SD7CX4OS2YCOBQKT57O3XV4E3LBUIGX4D6FQHQDSSZGWYKPA, seg_pos={0 0}, upload=536743]: Done uploading piece. err: <nil>
Dec 31 03:58:03.091834188 (612h29m2.962917233s) [seg_pos={0 0}, upload=536743]: Segment reached optimal threshold of 13 pieces.
Dec 31 03:58:03.092210124 (612h29m2.963293159s) [noise=true , address=145.79.227.33:4869 , piece_id=5E6ZGJJXNRPHBMXZY6T3R3NOSMYFK7MQAEJK75KIMQURI2WP5JYQ, seg_pos={0 0}, upload=536743]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CQxjq65r6xVTmNRurSQPzKxj4BoR6cJ8Uq29sTrdh6uXm5sq): protocol: expected piece hash; context canceled; EOF
Dec 31 03:58:03.092223550 (612h29m2.963306595s) [noise=true , address=145.79.227.33:4869 , piece_id=5E6ZGJJXNRPHBMXZY6T3R3NOSMYFK7MQAEJK75KIMQURI2WP5JYQ, seg_pos={0 0}, upload=536743]: Long tail cancellation due to optimal threshold
Dec 31 03:58:03.092239841 (612h29m2.963322886s) [noise=true , address=77.90.31.10:4518 , piece_id=C6CYAKMHVAYHK7OYFA672G4DBJXHAZ27SAFHXMZOBXW7CHSCI4QA, seg_pos={0 0}, upload=536743]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: expected piece hash; context canceled; EOF
Dec 31 03:58:03.092251062 (612h29m2.963334107s) [noise=true , address=77.90.31.10:4518 , piece_id=C6CYAKMHVAYHK7OYFA672G4DBJXHAZ27SAFHXMZOBXW7CHSCI4QA, seg_pos={0 0}, upload=536743]: Long tail cancellation due to optimal threshold
Dec 31 03:58:03.092492552 (612h29m2.963575587s) [noise=true , address=45.91.116.14:4191 , piece_id=WUDOWLIQEKSZY74HO7MQSRT7JE2LAWVFSZEA7VBT2M745VA4QJCA, seg_pos={0 0}, upload=536743]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: write tcp 188.166.225.170:57882->45.91.116.14:4191: use of closed network connection; write tcp 188.166.225.170:57882->45.91.116.14:4191: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57882->45.91.116.14:4191: use of closed network connection
Dec 31 03:58:03.092502310 (612h29m2.963585365s) [noise=true , address=45.91.116.14:4191 , piece_id=WUDOWLIQEKSZY74HO7MQSRT7JE2LAWVFSZEA7VBT2M745VA4QJCA, seg_pos={0 0}, upload=536743]: Long tail cancellation due to optimal threshold
Dec 31 03:58:03.092545584 (612h29m2.963628649s) [seg_pos={0 0}, upload=536743]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 31 03:58:03.092661704 (612h29m2.963744749s) [upload=536743]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 31 03:58:03.092668748 (612h29m2.963751793s) [upload=536743]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 31 03:58:03.382249155 (612h29m3.25333219s) [upload=536743]: Done uploading segments...
Dec 31 03:58:03.382285875 (612h29m3.25336891s) [upload=536743]: Upload finished. err: <nil>
Duration: 6.148745599s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [..............X.............................................] duration:
35.243855936s trace:
https://jaeger.dev.storj.io/trace/634c422994952cfc
FAIL (2025-12-31 03:58:37 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 121Xa3rvgKQhHc5NBufs57Ybnw3fYKUjgRyK32TTwWoi4E8m57o, Piece ID: 7ZJODSC7K4XWDIBJGMYHMGRFCXZGHTSGWTWPO7NKPWGGT73NFGOQ) context deadline exceeded; piecestore: (Node ID: 12caEiJobigXcGWwaX3Vt16zd2NXqk1WzLWxZME44E8Q8wdhn3q, Piece ID: RTZZG5ZDBPM74ZXITWZZ6LKJCKUOQ3YY4UUTMZ5VRVKFZE4YKAUA) context deadline exceeded; piecestore: (Node ID: 1zwuzCLWy2q166Ld52WUG9fvXNdJWsQrg2NearHYg7pyxpXFen, Piece ID: EJWBT3NCXTW5JSOIICAASNNDWFMIM2AO2SAE26S5POMMLEOYUH7Q) context deadline exceeded
======LOGS======
) [noise=true , address=193.149.164.143:4704 , piece_id=XIHZWOG63PEC3AQOHHWBLF53Y3B42NOO6TRNEUR3CIJB3YZTX3LQ, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.363237580 (611h51m51.234320615s) [noise=true , address=145.79.227.15:4889 , piece_id=EACS3IK7KTZI73UJEUOZB33TZN64WDWRC7TV5YJZ5ZMQVATQ3ORQ, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.364121825 (611h51m51.23520486s) [noise=true , address=193.149.164.143:4699 , piece_id=5LCUJBU4MEYEA7CQCA2FVVX6VCEX7SU3W4N3BBPQMLP6AHG5ZKKQ, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.364545251 (611h51m51.235628286s) [noise=true , address=45.91.116.11:4143 , piece_id=RTZZG5ZDBPM74ZXITWZZ6LKJCKUOQ3YY4UUTMZ5VRVKFZE4YKAUA, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.364959129 (611h51m51.236042174s) [noise=true , address=145.79.227.3:4176 , piece_id=TBPK43ULJ75RKDV5P63GXNVYZWG7TIQ3YLKMBLU55OQ2PFGSBH7A, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.365381323 (611h51m51.236464368s) [noise=true , address=145.79.227.44:4217 , piece_id=B5JSD2DUWSMSEME5CZCWFBXELK4WRMAXLLA2QQL73SB6WR6ECXUA, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.367027327 (611h51m51.238110362s) [noise=true , address=77.90.31.11:4561 , piece_id=IRE2JNK63DRH52RXBC5D3MRL6QO5S5N44UFVFWDRNY7LZBQ3IFMQ, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.367801231 (611h51m51.238884326s) [noise=true , address=145.79.227.40:4058 , piece_id=ECUZDPLHYQW3KYSTHKWBPD3CCTUCMAZ6XMOSH5S46ACF2T5366BQ, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.369086348 (611h51m51.240169404s) [noise=true , address=77.90.31.10:4536 , piece_id=EJWBT3NCXTW5JSOIICAASNNDWFMIM2AO2SAE26S5POMMLEOYUH7Q, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.370451017 (611h51m51.241534062s) [noise=true , address=77.90.31.19:4861 , piece_id=U7I2SDPMKDL4RLP2CIWZUSCEWPM5LP67VMUOLFKBHHVSCXL55MYQ, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.371980618 (611h51m51.243063654s) [noise=true , address=64.49.13.26:4955 , piece_id=53IGVDHKGGBIEKVJ6A52Q647E74KVI4FFRQU7VYK7POFU4CCX4HA, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.374009812 (611h51m51.245092847s) [noise=true , address=77.90.31.14:4671 , piece_id=2YXJ5RJERCNTD3TJHJHF4WBHR3DEOHDIOWI53H4PDS4EWQECEU2A, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.374500948 (611h51m51.245583993s) [noise=true , address=77.90.31.13:4632 , piece_id=72ADTLC2JC7Z3Z4GYKPECNAK6QMCMONGLUKZKQFLWN5UIQ4IPZYQ, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.392662589 (611h51m51.263745634s) [noise=true , address=145.79.227.32:4865 , piece_id=ZXAGYGEC35VVET2IAX2KJII4PUK3DFQXOFAYCC5R7KV5PNQCSUIA, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.393651513 (611h51m51.264734558s) [noise=true , address=145.79.227.31:4804 , piece_id=4E3PGEBPB2BFUAHNXJI4MZFWNNNZMMFDF5UBTT2UM4F2NHOHCIQQ, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.396266312 (611h51m51.267349357s) [noise=true , address=145.79.227.29:4966 , piece_id=N4JGQH23KY2BMVZZWWX7CKZN2FC2FJBFNAZEGZ7CRMRQAYEAZJLA, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:52.828565815 (611h51m52.69964886s) [upload=536207]: Next returned nil segment
Dec 31 03:20:52.828580594 (611h51m52.699663629s) [upload=536207]: Waiting for error group managing segments...
Dec 31 03:20:53.204430167 (611h51m53.075513202s) [noise=true , address=145.79.227.15:4889 , piece_id=EACS3IK7KTZI73UJEUOZB33TZN64WDWRC7TV5YJZ5ZMQVATQ3ORQ, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:53.265960889 (611h51m53.137043924s) [noise=true , address=145.79.227.31:4804 , piece_id=4E3PGEBPB2BFUAHNXJI4MZFWNNNZMMFDF5UBTT2UM4F2NHOHCIQQ, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:53.266193351 (611h51m53.137276386s) [noise=true , address=145.79.227.32:4865 , piece_id=ZXAGYGEC35VVET2IAX2KJII4PUK3DFQXOFAYCC5R7KV5PNQCSUIA, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:53.331763405 (611h51m53.20284644s) [noise=true , address=5.231.73.2:4035 , piece_id=7ZJODSC7K4XWDIBJGMYHMGRFCXZGHTSGWTWPO7NKPWGGT73NFGOQ, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:53.412602422 (611h51m53.283685467s) [noise=true , address=77.90.31.10:4536 , piece_id=EJWBT3NCXTW5JSOIICAASNNDWFMIM2AO2SAE26S5POMMLEOYUH7Q, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:53.419806240 (611h51m53.290889275s) [noise=true , address=77.90.31.14:4671 , piece_id=2YXJ5RJERCNTD3TJHJHF4WBHR3DEOHDIOWI53H4PDS4EWQECEU2A, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:53.498590626 (611h51m53.369673671s) [noise=true , address=77.90.31.11:4561 , piece_id=IRE2JNK63DRH52RXBC5D3MRL6QO5S5N44UFVFWDRNY7LZBQ3IFMQ, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:53.520983294 (611h51m53.392066339s) [noise=true , address=45.91.116.11:4143 , piece_id=RTZZG5ZDBPM74ZXITWZZ6LKJCKUOQ3YY4UUTMZ5VRVKFZE4YKAUA, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:53.585203506 (611h51m53.456286541s) [noise=true , address=77.90.31.19:4861 , piece_id=U7I2SDPMKDL4RLP2CIWZUSCEWPM5LP67VMUOLFKBHHVSCXL55MYQ, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:54.188950073 (611h51m54.060033119s) [noise=true , address=145.79.227.44:4217 , piece_id=B5JSD2DUWSMSEME5CZCWFBXELK4WRMAXLLA2QQL73SB6WR6ECXUA, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:54.204771008 (611h51m54.075854053s) [noise=true , address=145.79.227.40:4058 , piece_id=ECUZDPLHYQW3KYSTHKWBPD3CCTUCMAZ6XMOSH5S46ACF2T5366BQ, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:54.231988834 (611h51m54.103071879s) [noise=true , address=193.149.164.143:4699 , piece_id=5LCUJBU4MEYEA7CQCA2FVVX6VCEX7SU3W4N3BBPQMLP6AHG5ZKKQ, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:54.275280076 (611h51m54.146363121s) [noise=true , address=145.79.227.29:4966 , piece_id=N4JGQH23KY2BMVZZWWX7CKZN2FC2FJBFNAZEGZ7CRMRQAYEAZJLA, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:54.307143628 (611h51m54.178226673s) [noise=true , address=77.90.31.12:4602 , piece_id=ZPVIMHS5CQAKREW456SPNTHR3WM6IWZDB6AJZIYFREALJ6V63O6A, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:54.307168195 (611h51m54.17825124s) [seg_pos={0 0}, upload=536207]: Segment reached optimal threshold of 14 pieces.
Dec 31 03:20:54.307445734 (611h51m54.178528789s) [noise=true , address=77.90.31.13:4632 , piece_id=72ADTLC2JC7Z3Z4GYKPECNAK6QMCMONGLUKZKQFLWN5UIQ4IPZYQ, seg_pos={0 0}, upload=536207]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13iPed9zYrthvh36xo3k4gsqm4tz86iVkNvAr3wauuArQLquyS): protocol: expected piece hash; context canceled; EOF
Dec 31 03:20:54.307457145 (611h51m54.1785402s) [noise=true , address=77.90.31.13:4632 , piece_id=72ADTLC2JC7Z3Z4GYKPECNAK6QMCMONGLUKZKQFLWN5UIQ4IPZYQ, seg_pos={0 0}, upload=536207]: Long tail cancellation due to optimal threshold
Dec 31 03:20:54.307494436 (611h51m54.178577471s) [noise=true , address=193.149.164.143:4704 , piece_id=XIHZWOG63PEC3AQOHHWBLF53Y3B42NOO6TRNEUR3CIJB3YZTX3LQ, seg_pos={0 0}, upload=536207]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bUVmWsfAfQiPWhDkYRNGhims8TLQqGqvhrZMYugLLTdmLk2xY): protocol: expected piece hash; context canceled; EOF
Dec 31 03:20:54.307513032 (611h51m54.178596067s) [noise=true , address=193.149.164.143:4704 , piece_id=XIHZWOG63PEC3AQOHHWBLF53Y3B42NOO6TRNEUR3CIJB3YZTX3LQ, seg_pos={0 0}, upload=536207]: Long tail cancellation due to optimal threshold
Dec 31 03:20:54.307792985 (611h51m54.17887602s) [noise=true , address=145.79.227.3:4176 , piece_id=TBPK43ULJ75RKDV5P63GXNVYZWG7TIQ3YLKMBLU55OQ2PFGSBH7A, seg_pos={0 0}, upload=536207]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kGgrsNn5tv2GkcuVHHArCACwRB2qwbj3X3JmzVF4UL6dLRW1m): protocol: write tcp 188.166.225.170:39612->145.79.227.3:4176: use of closed network connection; write tcp 188.166.225.170:39612->145.79.227.3:4176: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39612->145.79.227.3:4176: use of closed network connection
Dec 31 03:20:54.307803214 (611h51m54.178886259s) [noise=true , address=145.79.227.3:4176 , piece_id=TBPK43ULJ75RKDV5P63GXNVYZWG7TIQ3YLKMBLU55OQ2PFGSBH7A, seg_pos={0 0}, upload=536207]: Long tail cancellation due to optimal threshold
Dec 31 03:20:54.307870673 (611h51m54.178953718s) [noise=true , address=64.49.13.26:4955 , piece_id=53IGVDHKGGBIEKVJ6A52Q647E74KVI4FFRQU7VYK7POFU4CCX4HA, seg_pos={0 0}, upload=536207]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125TV3v1CzV1Q9TPfyVHsn26NZ5XcHixKxA66t96iz62FF4SHAP): protocol: expected piece hash; context canceled; EOF
Dec 31 03:20:54.307882294 (611h51m54.178965339s) [noise=true , address=64.49.13.26:4955 , piece_id=53IGVDHKGGBIEKVJ6A52Q647E74KVI4FFRQU7VYK7POFU4CCX4HA, seg_pos={0 0}, upload=536207]: Long tail cancellation due to optimal threshold
Dec 31 03:20:54.307909116 (611h51m54.178992151s) [seg_pos={0 0}, upload=536207]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 31 03:20:54.308005189 (611h51m54.179088234s) [upload=536207]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 31 03:20:54.308013014 (611h51m54.179096049s) [upload=536207]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 31 03:20:54.593642638 (611h51m54.464725683s) [upload=536207]: Done uploading segments...
Dec 31 03:20:54.593663947 (611h51m54.464746983s) [upload=536207]: Upload finished. err: <nil>
Duration: 35.243950398s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [............X....X.X..X.....................................] duration:
7.114874238s trace:
https://jaeger.dev.storj.io/trace/5e7c34cde28aa313
FAIL (2025-12-31 03:58:36 UTC)
orbiter warning: context deadline exceeded
Duration: 7.115018092s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
24 out of 60 is failed (threshold=3) [.X.X.......X....X..X..X....X.XXXXX.X.....X.X...XXXX..XXXX..X] duration:
30.003623699s trace:
https://jaeger.dev.storj.io/trace/1c0826e28e8f88b4
FAIL (2025-12-31 03:58:29 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: ZTILSDDYFPH2WW64Y4636D5A3C6J4ULRRMZMZXFDCJ745A2FA7GA) context deadline exceeded; piecestore: (Node ID: 1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, Piece ID: CX7T7GQAZKFJDAYYRJ5CEXHPYQQMRMVEKBJRPTXI54H4DN4TMIYA) context deadline exceeded
======LOGS======
Dec 31 03:58:00.294001490 (612h29m0.165084525s) [upload=536743]: Starting upload
Dec 31 03:58:00.294080951 (612h29m0.165163996s) [upload=536743]: Done starting upload
Dec 31 03:58:00.294225086 (612h29m0.165308131s) [upload=536743]: Uploading segments...
Dec 31 03:58:00.294383307 (612h29m0.165466352s) [upload=536743]: Got next segment. Inline: false
Dec 31 03:58:00.294598156 (612h29m0.165681211s) [upload=536743]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 31 03:58:00.294606442 (612h29m0.165689487s) [upload=536743]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 31 03:58:00.542690384 (612h29m0.413773429s) [seg_pos={0 0}, upload=536743]: Begin upload segment...
Dec 31 03:58:00.542823247 (612h29m0.413906292s) [seg_pos={0 0}, upload=536743]: Done begin upload segment.
Dec 31 03:58:00.542914311 (612h29m0.413997356s) [noise=true , address=45.91.116.11:4119 , piece_id=FN44SD7CX4OS2YCOBQKT57O3XV4E3LBUIGX4D6FQHQDSSZGWYKPA, seg_pos={0 0}, upload=536743]: Uploading piece...
Dec 31 03:58:00.552138895 (612h29m0.42322194s) [noise=true , address=145.79.227.19:4077 , piece_id=D6QSNOT6P6643RMYKLT5ENAWCARLE37LQX4G6W23CTLZ3T4CTS5A, seg_pos={0 0}, upload=536743]: Uploading piece...
Dec 31 03:58:00.554471978 (612h29m0.425555013s) [noise=true , address=145.79.227.37:4566 , piece_id=O6PN2OLLMOISZ6XAJAEGG6XZ3M6646AAMU3L7X7KDC6MBSDLUGQA, seg_pos={0 0}, upload=536743]: Uploading piece...
Dec 31 03:58:00.554697908 (612h29m0.425780953s) [noise=true , address=45.91.116.6:4975 , piece_id=ZTILSDDYFPH2WW64Y4636D5A3C6J4ULRRMZMZXFDCJ745A2FA7GA, seg_pos={0 0}, upload=536743]: Uploading piece...
Dec 31 03:58:00.559476197 (612h29m0.430559242s) [noise=true , address=45.91.116.7:4009 , piece_id=CX7T7GQAZKFJDAYYRJ5CEXHPYQQMRMVEKBJRPTXI54H4DN4TMIYA, seg_pos={0 0}, upload=536743]: Uploading piece...
Dec 31 03:58:00.569351681 (612h29m0.440434726s) [noise=true , address=145.79.227.33:4869 , piece_id=5E6ZGJJXNRPHBMXZY6T3R3NOSMYFK7MQAEJK75KIMQURI2WP5JYQ, seg_pos={0 0}, upload=536743]: Uploading piece...
Dec 31 03:58:00.574259907 (612h29m0.445342952s) [noise=true , address=145.79.227.14:4839 , piece_id=XRHMRIBUKIJ4RG7LIMVDPPHGDWTC55TSEDDBKKMD6DJXIHNQ7RMQ, seg_pos={0 0}, upload=536743]: Uploading piece...
Dec 31 03:58:00.582068806 (612h29m0.453151851s) [noise=true , address=45.91.116.10:4083 , piece_id=SMTUKOKE6UJBFAWUFODMBYYTWNIBBXYAIYYCITBFKLH6LXXTMS3A, seg_pos={0 0}, upload=536743]: Uploading piece...
Dec 31 03:58:00.591056769 (612h29m0.462139845s) [noise=true , address=145.79.227.50:4373 , piece_id=Q4H7UMU2EEPUEAL57ITHQXRAFRK5BZXMNH6U7BGURPJAN4NCYMLA, seg_pos={0 0}, upload=536743]: Uploading piece...
Dec 31 03:58:00.601141192 (612h29m0.472224257s) [noise=true , address=45.91.116.12:4228 , piece_id=A2JARVBXJ4M3UCJLHAQIRBAHVY6L7OFAFI2LVDCZIXRM65QIWAVA, seg_pos={0 0}, upload=536743]: Uploading piece...
Dec 31 03:58:00.603482120 (612h29m0.474565165s) [noise=true , address=145.79.227.16:4909 , piece_id=R25YR4ZN6DQN7P3MQ5K6AOYOGYZDLV7TXM43NVHNCQSEQTYD6BSA, seg_pos={0 0}, upload=536743]: Uploading piece...
Dec 31 03:58:00.604129722 (612h29m0.475212757s) [noise=true , address=77.90.31.10:4518 , piece_id=C6CYAKMHVAYHK7OYFA672G4DBJXHAZ27SAFHXMZOBXW7CHSCI4QA, seg_pos={0 0}, upload=536743]: Uploading piece...
Dec 31 03:58:00.605368201 (612h29m0.476451246s) [noise=true , address=145.79.227.41:4085 , piece_id=XT3EU7WIT7FDTUC5J3Y5WI7CCSGH6ZAMU3BO5UJUDM2WYFOMLIVA, seg_pos={0 0}, upload=536743]: Uploading piece...
Dec 31 03:58:00.605539427 (612h29m0.476622462s) [noise=true , address=45.91.116.14:4191 , piece_id=WUDOWLIQEKSZY74HO7MQSRT7JE2LAWVFSZEA7VBT2M745VA4QJCA, seg_pos={0 0}, upload=536743]: Uploading piece...
Dec 31 03:58:00.607234183 (612h29m0.478317228s) [noise=true , address=145.79.227.15:4875 , piece_id=XTZBZPOCI7RK3O6LMPYLICJ4NY7WN7RNRHBWBZ7KKWIYFN2U77WA, seg_pos={0 0}, upload=536743]: Uploading piece...
Dec 31 03:58:00.608998473 (612h29m0.480081518s) [noise=true , address=145.79.227.31:4799 , piece_id=57TLZF7KTIETFEKNPW2XE4KATHNMJWF7ZB35I7CL3ZAANBOWYRSA, seg_pos={0 0}, upload=536743]: Uploading piece...
Dec 31 03:58:02.217068958 (612h29m2.088152004s) [upload=536743]: Next returned nil segment
Dec 31 03:58:02.217082103 (612h29m2.088165148s) [upload=536743]: Waiting for error group managing segments...
Dec 31 03:58:02.594337594 (612h29m2.465420639s) [noise=true , address=145.79.227.37:4566 , piece_id=O6PN2OLLMOISZ6XAJAEGG6XZ3M6646AAMU3L7X7KDC6MBSDLUGQA, seg_pos={0 0}, upload=536743]: Done uploading piece. err: <nil>
Dec 31 03:58:02.605087353 (612h29m2.476170398s) [noise=true , address=145.79.227.19:4077 , piece_id=D6QSNOT6P6643RMYKLT5ENAWCARLE37LQX4G6W23CTLZ3T4CTS5A, seg_pos={0 0}, upload=536743]: Done uploading piece. err: <nil>
Dec 31 03:58:02.607303622 (612h29m2.478386668s) [noise=true , address=145.79.227.16:4909 , piece_id=R25YR4ZN6DQN7P3MQ5K6AOYOGYZDLV7TXM43NVHNCQSEQTYD6BSA, seg_pos={0 0}, upload=536743]: Done uploading piece. err: <nil>
Dec 31 03:58:02.616067511 (612h29m2.487150546s) [noise=true , address=145.79.227.50:4373 , piece_id=Q4H7UMU2EEPUEAL57ITHQXRAFRK5BZXMNH6U7BGURPJAN4NCYMLA, seg_pos={0 0}, upload=536743]: Done uploading piece. err: <nil>
Dec 31 03:58:02.624055831 (612h29m2.495138876s) [noise=true , address=145.79.227.31:4799 , piece_id=57TLZF7KTIETFEKNPW2XE4KATHNMJWF7ZB35I7CL3ZAANBOWYRSA, seg_pos={0 0}, upload=536743]: Done uploading piece. err: <nil>
Dec 31 03:58:02.626248126 (612h29m2.497331171s) [noise=true , address=145.79.227.41:4085 , piece_id=XT3EU7WIT7FDTUC5J3Y5WI7CCSGH6ZAMU3BO5UJUDM2WYFOMLIVA, seg_pos={0 0}, upload=536743]: Done uploading piece. err: <nil>
Dec 31 03:58:02.673126778 (612h29m2.544209823s) [noise=true , address=145.79.227.14:4839 , piece_id=XRHMRIBUKIJ4RG7LIMVDPPHGDWTC55TSEDDBKKMD6DJXIHNQ7RMQ, seg_pos={0 0}, upload=536743]: Done uploading piece. err: <nil>
Dec 31 03:58:02.681227793 (612h29m2.552310838s) [noise=true , address=145.79.227.15:4875 , piece_id=XTZBZPOCI7RK3O6LMPYLICJ4NY7WN7RNRHBWBZ7KKWIYFN2U77WA, seg_pos={0 0}, upload=536743]: Done uploading piece. err: <nil>
Dec 31 03:58:02.902823439 (612h29m2.773906484s) [noise=true , address=45.91.116.6:4975 , piece_id=ZTILSDDYFPH2WW64Y4636D5A3C6J4ULRRMZMZXFDCJ745A2FA7GA, seg_pos={0 0}, upload=536743]: Done uploading piece. err: <nil>
Dec 31 03:58:02.924549789 (612h29m2.795632844s) [noise=true , address=45.91.116.12:4228 , piece_id=A2JARVBXJ4M3UCJLHAQIRBAHVY6L7OFAFI2LVDCZIXRM65QIWAVA, seg_pos={0 0}, upload=536743]: Done uploading piece. err: <nil>
Dec 31 03:58:02.937932651 (612h29m2.809015696s) [noise=true , address=45.91.116.10:4083 , piece_id=SMTUKOKE6UJBFAWUFODMBYYTWNIBBXYAIYYCITBFKLH6LXXTMS3A, seg_pos={0 0}, upload=536743]: Done uploading piece. err: <nil>
Dec 31 03:58:02.940111910 (612h29m2.811194956s) [noise=true , address=45.91.116.7:4009 , piece_id=CX7T7GQAZKFJDAYYRJ5CEXHPYQQMRMVEKBJRPTXI54H4DN4TMIYA, seg_pos={0 0}, upload=536743]: Done uploading piece. err: <nil>
Dec 31 03:58:03.091807317 (612h29m2.962890362s) [noise=true , address=45.91.116.11:4119 , piece_id=FN44SD7CX4OS2YCOBQKT57O3XV4E3LBUIGX4D6FQHQDSSZGWYKPA, seg_pos={0 0}, upload=536743]: Done uploading piece. err: <nil>
Dec 31 03:58:03.091834188 (612h29m2.962917233s) [seg_pos={0 0}, upload=536743]: Segment reached optimal threshold of 13 pieces.
Dec 31 03:58:03.092210124 (612h29m2.963293159s) [noise=true , address=145.79.227.33:4869 , piece_id=5E6ZGJJXNRPHBMXZY6T3R3NOSMYFK7MQAEJK75KIMQURI2WP5JYQ, seg_pos={0 0}, upload=536743]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CQxjq65r6xVTmNRurSQPzKxj4BoR6cJ8Uq29sTrdh6uXm5sq): protocol: expected piece hash; context canceled; EOF
Dec 31 03:58:03.092223550 (612h29m2.963306595s) [noise=true , address=145.79.227.33:4869 , piece_id=5E6ZGJJXNRPHBMXZY6T3R3NOSMYFK7MQAEJK75KIMQURI2WP5JYQ, seg_pos={0 0}, upload=536743]: Long tail cancellation due to optimal threshold
Dec 31 03:58:03.092239841 (612h29m2.963322886s) [noise=true , address=77.90.31.10:4518 , piece_id=C6CYAKMHVAYHK7OYFA672G4DBJXHAZ27SAFHXMZOBXW7CHSCI4QA, seg_pos={0 0}, upload=536743]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: expected piece hash; context canceled; EOF
Dec 31 03:58:03.092251062 (612h29m2.963334107s) [noise=true , address=77.90.31.10:4518 , piece_id=C6CYAKMHVAYHK7OYFA672G4DBJXHAZ27SAFHXMZOBXW7CHSCI4QA, seg_pos={0 0}, upload=536743]: Long tail cancellation due to optimal threshold
Dec 31 03:58:03.092492552 (612h29m2.963575587s) [noise=true , address=45.91.116.14:4191 , piece_id=WUDOWLIQEKSZY74HO7MQSRT7JE2LAWVFSZEA7VBT2M745VA4QJCA, seg_pos={0 0}, upload=536743]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: write tcp 188.166.225.170:57882->45.91.116.14:4191: use of closed network connection; write tcp 188.166.225.170:57882->45.91.116.14:4191: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57882->45.91.116.14:4191: use of closed network connection
Dec 31 03:58:03.092502310 (612h29m2.963585365s) [noise=true , address=45.91.116.14:4191 , piece_id=WUDOWLIQEKSZY74HO7MQSRT7JE2LAWVFSZEA7VBT2M745VA4QJCA, seg_pos={0 0}, upload=536743]: Long tail cancellation due to optimal threshold
Dec 31 03:58:03.092545584 (612h29m2.963628649s) [seg_pos={0 0}, upload=536743]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 31 03:58:03.092661704 (612h29m2.963744749s) [upload=536743]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 31 03:58:03.092668748 (612h29m2.963751793s) [upload=536743]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 31 03:58:03.382249155 (612h29m3.25333219s) [upload=536743]: Done uploading segments...
Dec 31 03:58:03.382285875 (612h29m3.25336891s) [upload=536743]: Upload finished. err: <nil>
Duration: 30.003648025s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [.............X....X.X..X....................................] duration:
19.740970056s trace:
https://jaeger.dev.storj.io/trace/5e7c34cde28aa313
FAIL (2025-12-31 03:58:12 UTC)
orbiter warning: context deadline exceeded
Duration: 19.740991556s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [...............X............................................] duration:
15.320489778s trace:
https://jaeger.dev.storj.io/trace/634c422994952cfc
FAIL (2025-12-31 03:58:01 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 121Xa3rvgKQhHc5NBufs57Ybnw3fYKUjgRyK32TTwWoi4E8m57o, Piece ID: 7ZJODSC7K4XWDIBJGMYHMGRFCXZGHTSGWTWPO7NKPWGGT73NFGOQ) context deadline exceeded; piecestore: (Node ID: 12caEiJobigXcGWwaX3Vt16zd2NXqk1WzLWxZME44E8Q8wdhn3q, Piece ID: RTZZG5ZDBPM74ZXITWZZ6LKJCKUOQ3YY4UUTMZ5VRVKFZE4YKAUA) context deadline exceeded; piecestore: (Node ID: 1zwuzCLWy2q166Ld52WUG9fvXNdJWsQrg2NearHYg7pyxpXFen, Piece ID: EJWBT3NCXTW5JSOIICAASNNDWFMIM2AO2SAE26S5POMMLEOYUH7Q) context deadline exceeded
======LOGS======
) [noise=true , address=193.149.164.143:4704 , piece_id=XIHZWOG63PEC3AQOHHWBLF53Y3B42NOO6TRNEUR3CIJB3YZTX3LQ, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.363237580 (611h51m51.234320615s) [noise=true , address=145.79.227.15:4889 , piece_id=EACS3IK7KTZI73UJEUOZB33TZN64WDWRC7TV5YJZ5ZMQVATQ3ORQ, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.364121825 (611h51m51.23520486s) [noise=true , address=193.149.164.143:4699 , piece_id=5LCUJBU4MEYEA7CQCA2FVVX6VCEX7SU3W4N3BBPQMLP6AHG5ZKKQ, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.364545251 (611h51m51.235628286s) [noise=true , address=45.91.116.11:4143 , piece_id=RTZZG5ZDBPM74ZXITWZZ6LKJCKUOQ3YY4UUTMZ5VRVKFZE4YKAUA, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.364959129 (611h51m51.236042174s) [noise=true , address=145.79.227.3:4176 , piece_id=TBPK43ULJ75RKDV5P63GXNVYZWG7TIQ3YLKMBLU55OQ2PFGSBH7A, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.365381323 (611h51m51.236464368s) [noise=true , address=145.79.227.44:4217 , piece_id=B5JSD2DUWSMSEME5CZCWFBXELK4WRMAXLLA2QQL73SB6WR6ECXUA, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.367027327 (611h51m51.238110362s) [noise=true , address=77.90.31.11:4561 , piece_id=IRE2JNK63DRH52RXBC5D3MRL6QO5S5N44UFVFWDRNY7LZBQ3IFMQ, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.367801231 (611h51m51.238884326s) [noise=true , address=145.79.227.40:4058 , piece_id=ECUZDPLHYQW3KYSTHKWBPD3CCTUCMAZ6XMOSH5S46ACF2T5366BQ, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.369086348 (611h51m51.240169404s) [noise=true , address=77.90.31.10:4536 , piece_id=EJWBT3NCXTW5JSOIICAASNNDWFMIM2AO2SAE26S5POMMLEOYUH7Q, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.370451017 (611h51m51.241534062s) [noise=true , address=77.90.31.19:4861 , piece_id=U7I2SDPMKDL4RLP2CIWZUSCEWPM5LP67VMUOLFKBHHVSCXL55MYQ, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.371980618 (611h51m51.243063654s) [noise=true , address=64.49.13.26:4955 , piece_id=53IGVDHKGGBIEKVJ6A52Q647E74KVI4FFRQU7VYK7POFU4CCX4HA, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.374009812 (611h51m51.245092847s) [noise=true , address=77.90.31.14:4671 , piece_id=2YXJ5RJERCNTD3TJHJHF4WBHR3DEOHDIOWI53H4PDS4EWQECEU2A, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.374500948 (611h51m51.245583993s) [noise=true , address=77.90.31.13:4632 , piece_id=72ADTLC2JC7Z3Z4GYKPECNAK6QMCMONGLUKZKQFLWN5UIQ4IPZYQ, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.392662589 (611h51m51.263745634s) [noise=true , address=145.79.227.32:4865 , piece_id=ZXAGYGEC35VVET2IAX2KJII4PUK3DFQXOFAYCC5R7KV5PNQCSUIA, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.393651513 (611h51m51.264734558s) [noise=true , address=145.79.227.31:4804 , piece_id=4E3PGEBPB2BFUAHNXJI4MZFWNNNZMMFDF5UBTT2UM4F2NHOHCIQQ, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.396266312 (611h51m51.267349357s) [noise=true , address=145.79.227.29:4966 , piece_id=N4JGQH23KY2BMVZZWWX7CKZN2FC2FJBFNAZEGZ7CRMRQAYEAZJLA, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:52.828565815 (611h51m52.69964886s) [upload=536207]: Next returned nil segment
Dec 31 03:20:52.828580594 (611h51m52.699663629s) [upload=536207]: Waiting for error group managing segments...
Dec 31 03:20:53.204430167 (611h51m53.075513202s) [noise=true , address=145.79.227.15:4889 , piece_id=EACS3IK7KTZI73UJEUOZB33TZN64WDWRC7TV5YJZ5ZMQVATQ3ORQ, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:53.265960889 (611h51m53.137043924s) [noise=true , address=145.79.227.31:4804 , piece_id=4E3PGEBPB2BFUAHNXJI4MZFWNNNZMMFDF5UBTT2UM4F2NHOHCIQQ, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:53.266193351 (611h51m53.137276386s) [noise=true , address=145.79.227.32:4865 , piece_id=ZXAGYGEC35VVET2IAX2KJII4PUK3DFQXOFAYCC5R7KV5PNQCSUIA, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:53.331763405 (611h51m53.20284644s) [noise=true , address=5.231.73.2:4035 , piece_id=7ZJODSC7K4XWDIBJGMYHMGRFCXZGHTSGWTWPO7NKPWGGT73NFGOQ, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:53.412602422 (611h51m53.283685467s) [noise=true , address=77.90.31.10:4536 , piece_id=EJWBT3NCXTW5JSOIICAASNNDWFMIM2AO2SAE26S5POMMLEOYUH7Q, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:53.419806240 (611h51m53.290889275s) [noise=true , address=77.90.31.14:4671 , piece_id=2YXJ5RJERCNTD3TJHJHF4WBHR3DEOHDIOWI53H4PDS4EWQECEU2A, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:53.498590626 (611h51m53.369673671s) [noise=true , address=77.90.31.11:4561 , piece_id=IRE2JNK63DRH52RXBC5D3MRL6QO5S5N44UFVFWDRNY7LZBQ3IFMQ, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:53.520983294 (611h51m53.392066339s) [noise=true , address=45.91.116.11:4143 , piece_id=RTZZG5ZDBPM74ZXITWZZ6LKJCKUOQ3YY4UUTMZ5VRVKFZE4YKAUA, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:53.585203506 (611h51m53.456286541s) [noise=true , address=77.90.31.19:4861 , piece_id=U7I2SDPMKDL4RLP2CIWZUSCEWPM5LP67VMUOLFKBHHVSCXL55MYQ, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:54.188950073 (611h51m54.060033119s) [noise=true , address=145.79.227.44:4217 , piece_id=B5JSD2DUWSMSEME5CZCWFBXELK4WRMAXLLA2QQL73SB6WR6ECXUA, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:54.204771008 (611h51m54.075854053s) [noise=true , address=145.79.227.40:4058 , piece_id=ECUZDPLHYQW3KYSTHKWBPD3CCTUCMAZ6XMOSH5S46ACF2T5366BQ, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:54.231988834 (611h51m54.103071879s) [noise=true , address=193.149.164.143:4699 , piece_id=5LCUJBU4MEYEA7CQCA2FVVX6VCEX7SU3W4N3BBPQMLP6AHG5ZKKQ, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:54.275280076 (611h51m54.146363121s) [noise=true , address=145.79.227.29:4966 , piece_id=N4JGQH23KY2BMVZZWWX7CKZN2FC2FJBFNAZEGZ7CRMRQAYEAZJLA, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:54.307143628 (611h51m54.178226673s) [noise=true , address=77.90.31.12:4602 , piece_id=ZPVIMHS5CQAKREW456SPNTHR3WM6IWZDB6AJZIYFREALJ6V63O6A, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:54.307168195 (611h51m54.17825124s) [seg_pos={0 0}, upload=536207]: Segment reached optimal threshold of 14 pieces.
Dec 31 03:20:54.307445734 (611h51m54.178528789s) [noise=true , address=77.90.31.13:4632 , piece_id=72ADTLC2JC7Z3Z4GYKPECNAK6QMCMONGLUKZKQFLWN5UIQ4IPZYQ, seg_pos={0 0}, upload=536207]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13iPed9zYrthvh36xo3k4gsqm4tz86iVkNvAr3wauuArQLquyS): protocol: expected piece hash; context canceled; EOF
Dec 31 03:20:54.307457145 (611h51m54.1785402s) [noise=true , address=77.90.31.13:4632 , piece_id=72ADTLC2JC7Z3Z4GYKPECNAK6QMCMONGLUKZKQFLWN5UIQ4IPZYQ, seg_pos={0 0}, upload=536207]: Long tail cancellation due to optimal threshold
Dec 31 03:20:54.307494436 (611h51m54.178577471s) [noise=true , address=193.149.164.143:4704 , piece_id=XIHZWOG63PEC3AQOHHWBLF53Y3B42NOO6TRNEUR3CIJB3YZTX3LQ, seg_pos={0 0}, upload=536207]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bUVmWsfAfQiPWhDkYRNGhims8TLQqGqvhrZMYugLLTdmLk2xY): protocol: expected piece hash; context canceled; EOF
Dec 31 03:20:54.307513032 (611h51m54.178596067s) [noise=true , address=193.149.164.143:4704 , piece_id=XIHZWOG63PEC3AQOHHWBLF53Y3B42NOO6TRNEUR3CIJB3YZTX3LQ, seg_pos={0 0}, upload=536207]: Long tail cancellation due to optimal threshold
Dec 31 03:20:54.307792985 (611h51m54.17887602s) [noise=true , address=145.79.227.3:4176 , piece_id=TBPK43ULJ75RKDV5P63GXNVYZWG7TIQ3YLKMBLU55OQ2PFGSBH7A, seg_pos={0 0}, upload=536207]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kGgrsNn5tv2GkcuVHHArCACwRB2qwbj3X3JmzVF4UL6dLRW1m): protocol: write tcp 188.166.225.170:39612->145.79.227.3:4176: use of closed network connection; write tcp 188.166.225.170:39612->145.79.227.3:4176: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39612->145.79.227.3:4176: use of closed network connection
Dec 31 03:20:54.307803214 (611h51m54.178886259s) [noise=true , address=145.79.227.3:4176 , piece_id=TBPK43ULJ75RKDV5P63GXNVYZWG7TIQ3YLKMBLU55OQ2PFGSBH7A, seg_pos={0 0}, upload=536207]: Long tail cancellation due to optimal threshold
Dec 31 03:20:54.307870673 (611h51m54.178953718s) [noise=true , address=64.49.13.26:4955 , piece_id=53IGVDHKGGBIEKVJ6A52Q647E74KVI4FFRQU7VYK7POFU4CCX4HA, seg_pos={0 0}, upload=536207]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125TV3v1CzV1Q9TPfyVHsn26NZ5XcHixKxA66t96iz62FF4SHAP): protocol: expected piece hash; context canceled; EOF
Dec 31 03:20:54.307882294 (611h51m54.178965339s) [noise=true , address=64.49.13.26:4955 , piece_id=53IGVDHKGGBIEKVJ6A52Q647E74KVI4FFRQU7VYK7POFU4CCX4HA, seg_pos={0 0}, upload=536207]: Long tail cancellation due to optimal threshold
Dec 31 03:20:54.307909116 (611h51m54.178992151s) [seg_pos={0 0}, upload=536207]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 31 03:20:54.308005189 (611h51m54.179088234s) [upload=536207]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 31 03:20:54.308013014 (611h51m54.179096049s) [upload=536207]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 31 03:20:54.593642638 (611h51m54.464725683s) [upload=536207]: Done uploading segments...
Dec 31 03:20:54.593663947 (611h51m54.464746983s) [upload=536207]: Upload finished. err: <nil>
Duration: 15.320517299s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
15 out of 60 is failed (threshold=6) [..X...X...X.X.X...X......X..XX.....X....X............XX.XX..] duration:
3.447868806s trace:
https://jaeger.dev.storj.io/trace/1bf049201ba7735a
FAIL (2025-12-31 03:58:00 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.447940193s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [................X...........................................] duration:
22.604962065s trace:
https://jaeger.dev.storj.io/trace/634c422994952cfc
FAIL (2025-12-31 03:57:45 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 121Xa3rvgKQhHc5NBufs57Ybnw3fYKUjgRyK32TTwWoi4E8m57o, Piece ID: 7ZJODSC7K4XWDIBJGMYHMGRFCXZGHTSGWTWPO7NKPWGGT73NFGOQ) context deadline exceeded; piecestore: (Node ID: 12caEiJobigXcGWwaX3Vt16zd2NXqk1WzLWxZME44E8Q8wdhn3q, Piece ID: RTZZG5ZDBPM74ZXITWZZ6LKJCKUOQ3YY4UUTMZ5VRVKFZE4YKAUA) context deadline exceeded; piecestore: (Node ID: 1zwuzCLWy2q166Ld52WUG9fvXNdJWsQrg2NearHYg7pyxpXFen, Piece ID: EJWBT3NCXTW5JSOIICAASNNDWFMIM2AO2SAE26S5POMMLEOYUH7Q) context deadline exceeded
======LOGS======
) [noise=true , address=193.149.164.143:4704 , piece_id=XIHZWOG63PEC3AQOHHWBLF53Y3B42NOO6TRNEUR3CIJB3YZTX3LQ, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.363237580 (611h51m51.234320615s) [noise=true , address=145.79.227.15:4889 , piece_id=EACS3IK7KTZI73UJEUOZB33TZN64WDWRC7TV5YJZ5ZMQVATQ3ORQ, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.364121825 (611h51m51.23520486s) [noise=true , address=193.149.164.143:4699 , piece_id=5LCUJBU4MEYEA7CQCA2FVVX6VCEX7SU3W4N3BBPQMLP6AHG5ZKKQ, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.364545251 (611h51m51.235628286s) [noise=true , address=45.91.116.11:4143 , piece_id=RTZZG5ZDBPM74ZXITWZZ6LKJCKUOQ3YY4UUTMZ5VRVKFZE4YKAUA, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.364959129 (611h51m51.236042174s) [noise=true , address=145.79.227.3:4176 , piece_id=TBPK43ULJ75RKDV5P63GXNVYZWG7TIQ3YLKMBLU55OQ2PFGSBH7A, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.365381323 (611h51m51.236464368s) [noise=true , address=145.79.227.44:4217 , piece_id=B5JSD2DUWSMSEME5CZCWFBXELK4WRMAXLLA2QQL73SB6WR6ECXUA, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.367027327 (611h51m51.238110362s) [noise=true , address=77.90.31.11:4561 , piece_id=IRE2JNK63DRH52RXBC5D3MRL6QO5S5N44UFVFWDRNY7LZBQ3IFMQ, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.367801231 (611h51m51.238884326s) [noise=true , address=145.79.227.40:4058 , piece_id=ECUZDPLHYQW3KYSTHKWBPD3CCTUCMAZ6XMOSH5S46ACF2T5366BQ, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.369086348 (611h51m51.240169404s) [noise=true , address=77.90.31.10:4536 , piece_id=EJWBT3NCXTW5JSOIICAASNNDWFMIM2AO2SAE26S5POMMLEOYUH7Q, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.370451017 (611h51m51.241534062s) [noise=true , address=77.90.31.19:4861 , piece_id=U7I2SDPMKDL4RLP2CIWZUSCEWPM5LP67VMUOLFKBHHVSCXL55MYQ, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.371980618 (611h51m51.243063654s) [noise=true , address=64.49.13.26:4955 , piece_id=53IGVDHKGGBIEKVJ6A52Q647E74KVI4FFRQU7VYK7POFU4CCX4HA, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.374009812 (611h51m51.245092847s) [noise=true , address=77.90.31.14:4671 , piece_id=2YXJ5RJERCNTD3TJHJHF4WBHR3DEOHDIOWI53H4PDS4EWQECEU2A, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.374500948 (611h51m51.245583993s) [noise=true , address=77.90.31.13:4632 , piece_id=72ADTLC2JC7Z3Z4GYKPECNAK6QMCMONGLUKZKQFLWN5UIQ4IPZYQ, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.392662589 (611h51m51.263745634s) [noise=true , address=145.79.227.32:4865 , piece_id=ZXAGYGEC35VVET2IAX2KJII4PUK3DFQXOFAYCC5R7KV5PNQCSUIA, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.393651513 (611h51m51.264734558s) [noise=true , address=145.79.227.31:4804 , piece_id=4E3PGEBPB2BFUAHNXJI4MZFWNNNZMMFDF5UBTT2UM4F2NHOHCIQQ, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.396266312 (611h51m51.267349357s) [noise=true , address=145.79.227.29:4966 , piece_id=N4JGQH23KY2BMVZZWWX7CKZN2FC2FJBFNAZEGZ7CRMRQAYEAZJLA, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:52.828565815 (611h51m52.69964886s) [upload=536207]: Next returned nil segment
Dec 31 03:20:52.828580594 (611h51m52.699663629s) [upload=536207]: Waiting for error group managing segments...
Dec 31 03:20:53.204430167 (611h51m53.075513202s) [noise=true , address=145.79.227.15:4889 , piece_id=EACS3IK7KTZI73UJEUOZB33TZN64WDWRC7TV5YJZ5ZMQVATQ3ORQ, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:53.265960889 (611h51m53.137043924s) [noise=true , address=145.79.227.31:4804 , piece_id=4E3PGEBPB2BFUAHNXJI4MZFWNNNZMMFDF5UBTT2UM4F2NHOHCIQQ, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:53.266193351 (611h51m53.137276386s) [noise=true , address=145.79.227.32:4865 , piece_id=ZXAGYGEC35VVET2IAX2KJII4PUK3DFQXOFAYCC5R7KV5PNQCSUIA, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:53.331763405 (611h51m53.20284644s) [noise=true , address=5.231.73.2:4035 , piece_id=7ZJODSC7K4XWDIBJGMYHMGRFCXZGHTSGWTWPO7NKPWGGT73NFGOQ, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:53.412602422 (611h51m53.283685467s) [noise=true , address=77.90.31.10:4536 , piece_id=EJWBT3NCXTW5JSOIICAASNNDWFMIM2AO2SAE26S5POMMLEOYUH7Q, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:53.419806240 (611h51m53.290889275s) [noise=true , address=77.90.31.14:4671 , piece_id=2YXJ5RJERCNTD3TJHJHF4WBHR3DEOHDIOWI53H4PDS4EWQECEU2A, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:53.498590626 (611h51m53.369673671s) [noise=true , address=77.90.31.11:4561 , piece_id=IRE2JNK63DRH52RXBC5D3MRL6QO5S5N44UFVFWDRNY7LZBQ3IFMQ, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:53.520983294 (611h51m53.392066339s) [noise=true , address=45.91.116.11:4143 , piece_id=RTZZG5ZDBPM74ZXITWZZ6LKJCKUOQ3YY4UUTMZ5VRVKFZE4YKAUA, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:53.585203506 (611h51m53.456286541s) [noise=true , address=77.90.31.19:4861 , piece_id=U7I2SDPMKDL4RLP2CIWZUSCEWPM5LP67VMUOLFKBHHVSCXL55MYQ, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:54.188950073 (611h51m54.060033119s) [noise=true , address=145.79.227.44:4217 , piece_id=B5JSD2DUWSMSEME5CZCWFBXELK4WRMAXLLA2QQL73SB6WR6ECXUA, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:54.204771008 (611h51m54.075854053s) [noise=true , address=145.79.227.40:4058 , piece_id=ECUZDPLHYQW3KYSTHKWBPD3CCTUCMAZ6XMOSH5S46ACF2T5366BQ, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:54.231988834 (611h51m54.103071879s) [noise=true , address=193.149.164.143:4699 , piece_id=5LCUJBU4MEYEA7CQCA2FVVX6VCEX7SU3W4N3BBPQMLP6AHG5ZKKQ, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:54.275280076 (611h51m54.146363121s) [noise=true , address=145.79.227.29:4966 , piece_id=N4JGQH23KY2BMVZZWWX7CKZN2FC2FJBFNAZEGZ7CRMRQAYEAZJLA, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:54.307143628 (611h51m54.178226673s) [noise=true , address=77.90.31.12:4602 , piece_id=ZPVIMHS5CQAKREW456SPNTHR3WM6IWZDB6AJZIYFREALJ6V63O6A, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:54.307168195 (611h51m54.17825124s) [seg_pos={0 0}, upload=536207]: Segment reached optimal threshold of 14 pieces.
Dec 31 03:20:54.307445734 (611h51m54.178528789s) [noise=true , address=77.90.31.13:4632 , piece_id=72ADTLC2JC7Z3Z4GYKPECNAK6QMCMONGLUKZKQFLWN5UIQ4IPZYQ, seg_pos={0 0}, upload=536207]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13iPed9zYrthvh36xo3k4gsqm4tz86iVkNvAr3wauuArQLquyS): protocol: expected piece hash; context canceled; EOF
Dec 31 03:20:54.307457145 (611h51m54.1785402s) [noise=true , address=77.90.31.13:4632 , piece_id=72ADTLC2JC7Z3Z4GYKPECNAK6QMCMONGLUKZKQFLWN5UIQ4IPZYQ, seg_pos={0 0}, upload=536207]: Long tail cancellation due to optimal threshold
Dec 31 03:20:54.307494436 (611h51m54.178577471s) [noise=true , address=193.149.164.143:4704 , piece_id=XIHZWOG63PEC3AQOHHWBLF53Y3B42NOO6TRNEUR3CIJB3YZTX3LQ, seg_pos={0 0}, upload=536207]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bUVmWsfAfQiPWhDkYRNGhims8TLQqGqvhrZMYugLLTdmLk2xY): protocol: expected piece hash; context canceled; EOF
Dec 31 03:20:54.307513032 (611h51m54.178596067s) [noise=true , address=193.149.164.143:4704 , piece_id=XIHZWOG63PEC3AQOHHWBLF53Y3B42NOO6TRNEUR3CIJB3YZTX3LQ, seg_pos={0 0}, upload=536207]: Long tail cancellation due to optimal threshold
Dec 31 03:20:54.307792985 (611h51m54.17887602s) [noise=true , address=145.79.227.3:4176 , piece_id=TBPK43ULJ75RKDV5P63GXNVYZWG7TIQ3YLKMBLU55OQ2PFGSBH7A, seg_pos={0 0}, upload=536207]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kGgrsNn5tv2GkcuVHHArCACwRB2qwbj3X3JmzVF4UL6dLRW1m): protocol: write tcp 188.166.225.170:39612->145.79.227.3:4176: use of closed network connection; write tcp 188.166.225.170:39612->145.79.227.3:4176: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39612->145.79.227.3:4176: use of closed network connection
Dec 31 03:20:54.307803214 (611h51m54.178886259s) [noise=true , address=145.79.227.3:4176 , piece_id=TBPK43ULJ75RKDV5P63GXNVYZWG7TIQ3YLKMBLU55OQ2PFGSBH7A, seg_pos={0 0}, upload=536207]: Long tail cancellation due to optimal threshold
Dec 31 03:20:54.307870673 (611h51m54.178953718s) [noise=true , address=64.49.13.26:4955 , piece_id=53IGVDHKGGBIEKVJ6A52Q647E74KVI4FFRQU7VYK7POFU4CCX4HA, seg_pos={0 0}, upload=536207]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125TV3v1CzV1Q9TPfyVHsn26NZ5XcHixKxA66t96iz62FF4SHAP): protocol: expected piece hash; context canceled; EOF
Dec 31 03:20:54.307882294 (611h51m54.178965339s) [noise=true , address=64.49.13.26:4955 , piece_id=53IGVDHKGGBIEKVJ6A52Q647E74KVI4FFRQU7VYK7POFU4CCX4HA, seg_pos={0 0}, upload=536207]: Long tail cancellation due to optimal threshold
Dec 31 03:20:54.307909116 (611h51m54.178992151s) [seg_pos={0 0}, upload=536207]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 31 03:20:54.308005189 (611h51m54.179088234s) [upload=536207]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 31 03:20:54.308013014 (611h51m54.179096049s) [upload=536207]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 31 03:20:54.593642638 (611h51m54.464725683s) [upload=536207]: Done uploading segments...
Dec 31 03:20:54.593663947 (611h51m54.464746983s) [upload=536207]: Upload finished. err: <nil>
Duration: 22.604986442s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [..............X....X.X..X...................................] duration:
7.243843612s trace:
https://jaeger.dev.storj.io/trace/5e7c34cde28aa313
FAIL (2025-12-31 03:57:29 UTC)
orbiter warning: context deadline exceeded
Duration: 7.243867129s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [...............X....X.X..X..................................] duration:
11.076984058s trace:
https://jaeger.dev.storj.io/trace/5e7c34cde28aa313
FAIL (2025-12-31 03:57:05 UTC)
orbiter warning: context deadline exceeded
Duration: 11.077010508s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [.................X..........................................] duration:
29.451327327s trace:
https://jaeger.dev.storj.io/trace/634c422994952cfc
FAIL (2025-12-31 03:57:00 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 121Xa3rvgKQhHc5NBufs57Ybnw3fYKUjgRyK32TTwWoi4E8m57o, Piece ID: 7ZJODSC7K4XWDIBJGMYHMGRFCXZGHTSGWTWPO7NKPWGGT73NFGOQ) context deadline exceeded; piecestore: (Node ID: 12caEiJobigXcGWwaX3Vt16zd2NXqk1WzLWxZME44E8Q8wdhn3q, Piece ID: RTZZG5ZDBPM74ZXITWZZ6LKJCKUOQ3YY4UUTMZ5VRVKFZE4YKAUA) context deadline exceeded; piecestore: (Node ID: 1zwuzCLWy2q166Ld52WUG9fvXNdJWsQrg2NearHYg7pyxpXFen, Piece ID: EJWBT3NCXTW5JSOIICAASNNDWFMIM2AO2SAE26S5POMMLEOYUH7Q) context deadline exceeded
======LOGS======
) [noise=true , address=193.149.164.143:4704 , piece_id=XIHZWOG63PEC3AQOHHWBLF53Y3B42NOO6TRNEUR3CIJB3YZTX3LQ, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.363237580 (611h51m51.234320615s) [noise=true , address=145.79.227.15:4889 , piece_id=EACS3IK7KTZI73UJEUOZB33TZN64WDWRC7TV5YJZ5ZMQVATQ3ORQ, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.364121825 (611h51m51.23520486s) [noise=true , address=193.149.164.143:4699 , piece_id=5LCUJBU4MEYEA7CQCA2FVVX6VCEX7SU3W4N3BBPQMLP6AHG5ZKKQ, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.364545251 (611h51m51.235628286s) [noise=true , address=45.91.116.11:4143 , piece_id=RTZZG5ZDBPM74ZXITWZZ6LKJCKUOQ3YY4UUTMZ5VRVKFZE4YKAUA, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.364959129 (611h51m51.236042174s) [noise=true , address=145.79.227.3:4176 , piece_id=TBPK43ULJ75RKDV5P63GXNVYZWG7TIQ3YLKMBLU55OQ2PFGSBH7A, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.365381323 (611h51m51.236464368s) [noise=true , address=145.79.227.44:4217 , piece_id=B5JSD2DUWSMSEME5CZCWFBXELK4WRMAXLLA2QQL73SB6WR6ECXUA, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.367027327 (611h51m51.238110362s) [noise=true , address=77.90.31.11:4561 , piece_id=IRE2JNK63DRH52RXBC5D3MRL6QO5S5N44UFVFWDRNY7LZBQ3IFMQ, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.367801231 (611h51m51.238884326s) [noise=true , address=145.79.227.40:4058 , piece_id=ECUZDPLHYQW3KYSTHKWBPD3CCTUCMAZ6XMOSH5S46ACF2T5366BQ, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.369086348 (611h51m51.240169404s) [noise=true , address=77.90.31.10:4536 , piece_id=EJWBT3NCXTW5JSOIICAASNNDWFMIM2AO2SAE26S5POMMLEOYUH7Q, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.370451017 (611h51m51.241534062s) [noise=true , address=77.90.31.19:4861 , piece_id=U7I2SDPMKDL4RLP2CIWZUSCEWPM5LP67VMUOLFKBHHVSCXL55MYQ, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.371980618 (611h51m51.243063654s) [noise=true , address=64.49.13.26:4955 , piece_id=53IGVDHKGGBIEKVJ6A52Q647E74KVI4FFRQU7VYK7POFU4CCX4HA, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.374009812 (611h51m51.245092847s) [noise=true , address=77.90.31.14:4671 , piece_id=2YXJ5RJERCNTD3TJHJHF4WBHR3DEOHDIOWI53H4PDS4EWQECEU2A, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.374500948 (611h51m51.245583993s) [noise=true , address=77.90.31.13:4632 , piece_id=72ADTLC2JC7Z3Z4GYKPECNAK6QMCMONGLUKZKQFLWN5UIQ4IPZYQ, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.392662589 (611h51m51.263745634s) [noise=true , address=145.79.227.32:4865 , piece_id=ZXAGYGEC35VVET2IAX2KJII4PUK3DFQXOFAYCC5R7KV5PNQCSUIA, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.393651513 (611h51m51.264734558s) [noise=true , address=145.79.227.31:4804 , piece_id=4E3PGEBPB2BFUAHNXJI4MZFWNNNZMMFDF5UBTT2UM4F2NHOHCIQQ, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.396266312 (611h51m51.267349357s) [noise=true , address=145.79.227.29:4966 , piece_id=N4JGQH23KY2BMVZZWWX7CKZN2FC2FJBFNAZEGZ7CRMRQAYEAZJLA, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:52.828565815 (611h51m52.69964886s) [upload=536207]: Next returned nil segment
Dec 31 03:20:52.828580594 (611h51m52.699663629s) [upload=536207]: Waiting for error group managing segments...
Dec 31 03:20:53.204430167 (611h51m53.075513202s) [noise=true , address=145.79.227.15:4889 , piece_id=EACS3IK7KTZI73UJEUOZB33TZN64WDWRC7TV5YJZ5ZMQVATQ3ORQ, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:53.265960889 (611h51m53.137043924s) [noise=true , address=145.79.227.31:4804 , piece_id=4E3PGEBPB2BFUAHNXJI4MZFWNNNZMMFDF5UBTT2UM4F2NHOHCIQQ, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:53.266193351 (611h51m53.137276386s) [noise=true , address=145.79.227.32:4865 , piece_id=ZXAGYGEC35VVET2IAX2KJII4PUK3DFQXOFAYCC5R7KV5PNQCSUIA, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:53.331763405 (611h51m53.20284644s) [noise=true , address=5.231.73.2:4035 , piece_id=7ZJODSC7K4XWDIBJGMYHMGRFCXZGHTSGWTWPO7NKPWGGT73NFGOQ, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:53.412602422 (611h51m53.283685467s) [noise=true , address=77.90.31.10:4536 , piece_id=EJWBT3NCXTW5JSOIICAASNNDWFMIM2AO2SAE26S5POMMLEOYUH7Q, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:53.419806240 (611h51m53.290889275s) [noise=true , address=77.90.31.14:4671 , piece_id=2YXJ5RJERCNTD3TJHJHF4WBHR3DEOHDIOWI53H4PDS4EWQECEU2A, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:53.498590626 (611h51m53.369673671s) [noise=true , address=77.90.31.11:4561 , piece_id=IRE2JNK63DRH52RXBC5D3MRL6QO5S5N44UFVFWDRNY7LZBQ3IFMQ, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:53.520983294 (611h51m53.392066339s) [noise=true , address=45.91.116.11:4143 , piece_id=RTZZG5ZDBPM74ZXITWZZ6LKJCKUOQ3YY4UUTMZ5VRVKFZE4YKAUA, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:53.585203506 (611h51m53.456286541s) [noise=true , address=77.90.31.19:4861 , piece_id=U7I2SDPMKDL4RLP2CIWZUSCEWPM5LP67VMUOLFKBHHVSCXL55MYQ, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:54.188950073 (611h51m54.060033119s) [noise=true , address=145.79.227.44:4217 , piece_id=B5JSD2DUWSMSEME5CZCWFBXELK4WRMAXLLA2QQL73SB6WR6ECXUA, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:54.204771008 (611h51m54.075854053s) [noise=true , address=145.79.227.40:4058 , piece_id=ECUZDPLHYQW3KYSTHKWBPD3CCTUCMAZ6XMOSH5S46ACF2T5366BQ, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:54.231988834 (611h51m54.103071879s) [noise=true , address=193.149.164.143:4699 , piece_id=5LCUJBU4MEYEA7CQCA2FVVX6VCEX7SU3W4N3BBPQMLP6AHG5ZKKQ, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:54.275280076 (611h51m54.146363121s) [noise=true , address=145.79.227.29:4966 , piece_id=N4JGQH23KY2BMVZZWWX7CKZN2FC2FJBFNAZEGZ7CRMRQAYEAZJLA, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:54.307143628 (611h51m54.178226673s) [noise=true , address=77.90.31.12:4602 , piece_id=ZPVIMHS5CQAKREW456SPNTHR3WM6IWZDB6AJZIYFREALJ6V63O6A, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:54.307168195 (611h51m54.17825124s) [seg_pos={0 0}, upload=536207]: Segment reached optimal threshold of 14 pieces.
Dec 31 03:20:54.307445734 (611h51m54.178528789s) [noise=true , address=77.90.31.13:4632 , piece_id=72ADTLC2JC7Z3Z4GYKPECNAK6QMCMONGLUKZKQFLWN5UIQ4IPZYQ, seg_pos={0 0}, upload=536207]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13iPed9zYrthvh36xo3k4gsqm4tz86iVkNvAr3wauuArQLquyS): protocol: expected piece hash; context canceled; EOF
Dec 31 03:20:54.307457145 (611h51m54.1785402s) [noise=true , address=77.90.31.13:4632 , piece_id=72ADTLC2JC7Z3Z4GYKPECNAK6QMCMONGLUKZKQFLWN5UIQ4IPZYQ, seg_pos={0 0}, upload=536207]: Long tail cancellation due to optimal threshold
Dec 31 03:20:54.307494436 (611h51m54.178577471s) [noise=true , address=193.149.164.143:4704 , piece_id=XIHZWOG63PEC3AQOHHWBLF53Y3B42NOO6TRNEUR3CIJB3YZTX3LQ, seg_pos={0 0}, upload=536207]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bUVmWsfAfQiPWhDkYRNGhims8TLQqGqvhrZMYugLLTdmLk2xY): protocol: expected piece hash; context canceled; EOF
Dec 31 03:20:54.307513032 (611h51m54.178596067s) [noise=true , address=193.149.164.143:4704 , piece_id=XIHZWOG63PEC3AQOHHWBLF53Y3B42NOO6TRNEUR3CIJB3YZTX3LQ, seg_pos={0 0}, upload=536207]: Long tail cancellation due to optimal threshold
Dec 31 03:20:54.307792985 (611h51m54.17887602s) [noise=true , address=145.79.227.3:4176 , piece_id=TBPK43ULJ75RKDV5P63GXNVYZWG7TIQ3YLKMBLU55OQ2PFGSBH7A, seg_pos={0 0}, upload=536207]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kGgrsNn5tv2GkcuVHHArCACwRB2qwbj3X3JmzVF4UL6dLRW1m): protocol: write tcp 188.166.225.170:39612->145.79.227.3:4176: use of closed network connection; write tcp 188.166.225.170:39612->145.79.227.3:4176: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39612->145.79.227.3:4176: use of closed network connection
Dec 31 03:20:54.307803214 (611h51m54.178886259s) [noise=true , address=145.79.227.3:4176 , piece_id=TBPK43ULJ75RKDV5P63GXNVYZWG7TIQ3YLKMBLU55OQ2PFGSBH7A, seg_pos={0 0}, upload=536207]: Long tail cancellation due to optimal threshold
Dec 31 03:20:54.307870673 (611h51m54.178953718s) [noise=true , address=64.49.13.26:4955 , piece_id=53IGVDHKGGBIEKVJ6A52Q647E74KVI4FFRQU7VYK7POFU4CCX4HA, seg_pos={0 0}, upload=536207]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125TV3v1CzV1Q9TPfyVHsn26NZ5XcHixKxA66t96iz62FF4SHAP): protocol: expected piece hash; context canceled; EOF
Dec 31 03:20:54.307882294 (611h51m54.178965339s) [noise=true , address=64.49.13.26:4955 , piece_id=53IGVDHKGGBIEKVJ6A52Q647E74KVI4FFRQU7VYK7POFU4CCX4HA, seg_pos={0 0}, upload=536207]: Long tail cancellation due to optimal threshold
Dec 31 03:20:54.307909116 (611h51m54.178992151s) [seg_pos={0 0}, upload=536207]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 31 03:20:54.308005189 (611h51m54.179088234s) [upload=536207]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 31 03:20:54.308013014 (611h51m54.179096049s) [upload=536207]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 31 03:20:54.593642638 (611h51m54.464725683s) [upload=536207]: Done uploading segments...
Dec 31 03:20:54.593663947 (611h51m54.464746983s) [upload=536207]: Upload finished. err: <nil>
Duration: 29.451355322s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
15 out of 60 is failed (threshold=6) [...X...X...X.X.X...X......X..XX.....X....X............XX.XX.] duration:
3.675971485s trace:
https://jaeger.dev.storj.io/trace/1bf049201ba7735a
FAIL (2025-12-31 03:56:55 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.676009258s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
23 out of 60 is failed (threshold=3) [..X.X.......X....X..X..X....X.XXXXX.X.....X.X...XXXX..XXXX..] duration:
6.716875904s trace:
https://jaeger.dev.storj.io/trace/6682252d7557c0f0
FAIL (2025-12-31 03:56:54 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: QT2ROG3IISNYE4QWIX7XFQH4CMTL6OT256X4IZP36O3U7DFJTNKQ) context deadline exceeded; piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: 4VON2DQN4YUE6OHKDMMIUIPM6VBA57F34UBAIEIAEGSDNU3AZEPA) context deadline exceeded; piecestore: (Node ID: 1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb, Piece ID: E4JIAWYCJF2QWHZRMMJCNDVWPG5SDJKF2RXXDVG7NMEEFTJZJXZA) context deadline exceeded
======LOGS======
Dec 31 03:55:29.610844489 (612h26m29.481927524s) [upload=536708]: Starting upload
Dec 31 03:55:29.610905506 (612h26m29.481988551s) [upload=536708]: Done starting upload
Dec 31 03:55:29.611054219 (612h26m29.482137254s) [upload=536708]: Uploading segments...
Dec 31 03:55:29.611720016 (612h26m29.482803061s) [upload=536708]: Got next segment. Inline: false
Dec 31 03:55:29.611774230 (612h26m29.482857265s) [upload=536708]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 31 03:55:29.611780622 (612h26m29.482863657s) [upload=536708]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 31 03:55:29.859696415 (612h26m29.73077945s) [seg_pos={0 0}, upload=536708]: Begin upload segment...
Dec 31 03:55:29.859753674 (612h26m29.730836719s) [seg_pos={0 0}, upload=536708]: Done begin upload segment.
Dec 31 03:55:29.859793790 (612h26m29.730876825s) [noise=true , address=77.90.31.21:4880 , piece_id=E4JIAWYCJF2QWHZRMMJCNDVWPG5SDJKF2RXXDVG7NMEEFTJZJXZA, seg_pos={0 0}, upload=536708]: Uploading piece...
Dec 31 03:55:29.859809850 (612h26m29.730892895s) [noise=true , address=145.79.227.19:4077 , piece_id=W4T3ATYZAI3YV6WD6UC3AXDGARHMC3HRXGXS25YBJSKLXD3JMV5Q, seg_pos={0 0}, upload=536708]: Uploading piece...
Dec 31 03:55:29.860369586 (612h26m29.731452621s) [noise=true , address=145.79.227.27:4690 , piece_id=5AWAMERXMY5LQH4HXEVTZ5JOPN5Z34HWPTSJKKDCDQYUFOXRIWDA, seg_pos={0 0}, upload=536708]: Uploading piece...
Dec 31 03:55:29.860847637 (612h26m29.731930672s) [noise=true , address=145.79.227.21:4474 , piece_id=TFDUOMBP474RIWGRHWLSL7TNOHNP7Q44Q6ADJKEB3DY54HQ5PRFQ, seg_pos={0 0}, upload=536708]: Uploading piece...
Dec 31 03:55:29.861240946 (612h26m29.732323981s) [noise=true , address=145.79.227.34:4905 , piece_id=UXSGKRZ66C5I2JT4PAHWC5F35P34TFBJC623PMCUY5WE2W4OSEMA, seg_pos={0 0}, upload=536708]: Uploading piece...
Dec 31 03:55:29.861247649 (612h26m29.732330694s) [noise=true , address=145.79.227.49:4337 , piece_id=VVUK7EQMF4XGVFMEBQBIPSTX5FBNMOAORY7DJUD4SMZCREBK7ANQ, seg_pos={0 0}, upload=536708]: Uploading piece...
Dec 31 03:55:29.861597725 (612h26m29.73268076s) [noise=true , address=77.90.31.11:4554 , piece_id=PKQ6I5WRTYV5MTOPMQON5ESRI4GT5KTFHSZZHHVYEYVLPW4BBG3Q, seg_pos={0 0}, upload=536708]: Uploading piece...
Dec 31 03:55:29.861846459 (612h26m29.732929494s) [noise=true , address=145.79.227.3:4149 , piece_id=JLV7CFVNSXDJFCKPCEE27UCSH5YVCTZJGOYFSQEDIO23IYM6QEOA, seg_pos={0 0}, upload=536708]: Uploading piece...
Dec 31 03:55:29.862245879 (612h26m29.733328914s) [noise=true , address=145.79.227.50:4375 , piece_id=5XEPQHLC2FKJIJGQXY7KH4B5ESOVUNEBTLFAQRQB6HJRWBVPHTDQ, seg_pos={0 0}, upload=536708]: Uploading piece...
Dec 31 03:55:29.862675798 (612h26m29.733758833s) [noise=true , address=77.90.31.10:4519 , piece_id=QT2ROG3IISNYE4QWIX7XFQH4CMTL6OT256X4IZP36O3U7DFJTNKQ, seg_pos={0 0}, upload=536708]: Uploading piece...
Dec 31 03:55:29.863048678 (612h26m29.734131713s) [noise=true , address=145.79.227.40:4049 , piece_id=3GVHPGMLAUUVRFEHO5BGP7UWCIQHCQ67RPK7TQV4BNEPFGG4ZSWA, seg_pos={0 0}, upload=536708]: Uploading piece...
Dec 31 03:55:29.863401139 (612h26m29.734484184s) [noise=true , address=145.79.227.52:4445 , piece_id=XDFC2E4K7P6ZP3T3QKFJOW6BNOJ5IGSRUX2W54HSG3F3FRGK5BUQ, seg_pos={0 0}, upload=536708]: Uploading piece...
Dec 31 03:55:29.863830968 (612h26m29.734914003s) [noise=true , address=45.91.116.9:4263 , piece_id=IQGT7QAUZUGTRRC7NL3IDUESBQANMOTGXN26KNBCXWBPUA6LFWBQ, seg_pos={0 0}, upload=536708]: Uploading piece...
Dec 31 03:55:29.864208145 (612h26m29.73529119s) [noise=true , address=145.79.227.42:4121 , piece_id=RZALVHACRGXFDA2TSN7M4KOSNMTVV6HF5VN56AVQBSLP57UDVYUA, seg_pos={0 0}, upload=536708]: Uploading piece...
Dec 31 03:55:29.864315971 (612h26m29.735399006s) [noise=true , address=45.91.116.13:4154 , piece_id=UMUMUAKI6DVU7E2URNXPX2HV7CA2QQMMFADPEF6KH6QKDNHPHNOA, seg_pos={0 0}, upload=536708]: Uploading piece...
Dec 31 03:55:29.864838185 (612h26m29.73592122s) [noise=true , address=45.91.116.6:4975 , piece_id=4VON2DQN4YUE6OHKDMMIUIPM6VBA57F34UBAIEIAEGSDNU3AZEPA, seg_pos={0 0}, upload=536708]: Uploading piece...
Dec 31 03:55:31.563613463 (612h26m31.434696498s) [upload=536708]: Next returned nil segment
Dec 31 03:55:31.563628091 (612h26m31.434711126s) [upload=536708]: Waiting for error group managing segments...
Dec 31 03:55:31.850844927 (612h26m31.721927972s) [noise=true , address=145.79.227.19:4077 , piece_id=W4T3ATYZAI3YV6WD6UC3AXDGARHMC3HRXGXS25YBJSKLXD3JMV5Q, seg_pos={0 0}, upload=536708]: Done uploading piece. err: <nil>
Dec 31 03:55:31.868260779 (612h26m31.739343825s) [noise=true , address=145.79.227.27:4690 , piece_id=5AWAMERXMY5LQH4HXEVTZ5JOPN5Z34HWPTSJKKDCDQYUFOXRIWDA, seg_pos={0 0}, upload=536708]: Done uploading piece. err: <nil>
Dec 31 03:55:31.872717676 (612h26m31.743800711s) [noise=true , address=145.79.227.52:4445 , piece_id=XDFC2E4K7P6ZP3T3QKFJOW6BNOJ5IGSRUX2W54HSG3F3FRGK5BUQ, seg_pos={0 0}, upload=536708]: Done uploading piece. err: <nil>
Dec 31 03:55:31.879399198 (612h26m31.750482243s) [noise=true , address=145.79.227.50:4375 , piece_id=5XEPQHLC2FKJIJGQXY7KH4B5ESOVUNEBTLFAQRQB6HJRWBVPHTDQ, seg_pos={0 0}, upload=536708]: Done uploading piece. err: <nil>
Dec 31 03:55:31.914183161 (612h26m31.785266196s) [noise=true , address=145.79.227.21:4474 , piece_id=TFDUOMBP474RIWGRHWLSL7TNOHNP7Q44Q6ADJKEB3DY54HQ5PRFQ, seg_pos={0 0}, upload=536708]: Done uploading piece. err: <nil>
Dec 31 03:55:31.932902375 (612h26m31.80398542s) [noise=true , address=145.79.227.42:4121 , piece_id=RZALVHACRGXFDA2TSN7M4KOSNMTVV6HF5VN56AVQBSLP57UDVYUA, seg_pos={0 0}, upload=536708]: Done uploading piece. err: <nil>
Dec 31 03:55:32.101017818 (612h26m31.972100853s) [noise=true , address=77.90.31.11:4554 , piece_id=PKQ6I5WRTYV5MTOPMQON5ESRI4GT5KTFHSZZHHVYEYVLPW4BBG3Q, seg_pos={0 0}, upload=536708]: Done uploading piece. err: <nil>
Dec 31 03:55:32.116944784 (612h26m31.988027829s) [noise=true , address=145.79.227.3:4149 , piece_id=JLV7CFVNSXDJFCKPCEE27UCSH5YVCTZJGOYFSQEDIO23IYM6QEOA, seg_pos={0 0}, upload=536708]: Done uploading piece. err: <nil>
Dec 31 03:55:32.135135732 (612h26m32.006218767s) [noise=true , address=77.90.31.21:4880 , piece_id=E4JIAWYCJF2QWHZRMMJCNDVWPG5SDJKF2RXXDVG7NMEEFTJZJXZA, seg_pos={0 0}, upload=536708]: Done uploading piece. err: <nil>
Dec 31 03:55:32.149722257 (612h26m32.020805292s) [noise=true , address=77.90.31.10:4519 , piece_id=QT2ROG3IISNYE4QWIX7XFQH4CMTL6OT256X4IZP36O3U7DFJTNKQ, seg_pos={0 0}, upload=536708]: Done uploading piece. err: <nil>
Dec 31 03:55:32.200260328 (612h26m32.071343373s) [noise=true , address=45.91.116.9:4263 , piece_id=IQGT7QAUZUGTRRC7NL3IDUESBQANMOTGXN26KNBCXWBPUA6LFWBQ, seg_pos={0 0}, upload=536708]: Done uploading piece. err: <nil>
Dec 31 03:55:32.224638697 (612h26m32.095721742s) [noise=true , address=45.91.116.6:4975 , piece_id=4VON2DQN4YUE6OHKDMMIUIPM6VBA57F34UBAIEIAEGSDNU3AZEPA, seg_pos={0 0}, upload=536708]: Done uploading piece. err: <nil>
Dec 31 03:55:32.464758784 (612h26m32.335841819s) [noise=true , address=45.91.116.13:4154 , piece_id=UMUMUAKI6DVU7E2URNXPX2HV7CA2QQMMFADPEF6KH6QKDNHPHNOA, seg_pos={0 0}, upload=536708]: Done uploading piece. err: <nil>
Dec 31 03:55:32.464778162 (612h26m32.335861197s) [seg_pos={0 0}, upload=536708]: Segment reached optimal threshold of 13 pieces.
Dec 31 03:55:32.464964857 (612h26m32.336047892s) [noise=true , address=145.79.227.40:4049 , piece_id=3GVHPGMLAUUVRFEHO5BGP7UWCIQHCQ67RPK7TQV4BNEPFGG4ZSWA, seg_pos={0 0}, upload=536708]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J): protocol: expected piece hash; context canceled; EOF
Dec 31 03:55:32.464974164 (612h26m32.336057199s) [noise=true , address=145.79.227.40:4049 , piece_id=3GVHPGMLAUUVRFEHO5BGP7UWCIQHCQ67RPK7TQV4BNEPFGG4ZSWA, seg_pos={0 0}, upload=536708]: Long tail cancellation due to optimal threshold
Dec 31 03:55:32.465155780 (612h26m32.336238815s) [noise=true , address=145.79.227.49:4337 , piece_id=VVUK7EQMF4XGVFMEBQBIPSTX5FBNMOAORY7DJUD4SMZCREBK7ANQ, seg_pos={0 0}, upload=536708]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nTZFfzNCcVAjut7NXf9seifysJDfvXkS8aZSnXKfw6gh1TrjE): protocol: expected piece hash; context canceled; EOF
Dec 31 03:55:32.465162413 (612h26m32.336245448s) [noise=true , address=145.79.227.49:4337 , piece_id=VVUK7EQMF4XGVFMEBQBIPSTX5FBNMOAORY7DJUD4SMZCREBK7ANQ, seg_pos={0 0}, upload=536708]: Long tail cancellation due to optimal threshold
Dec 31 03:55:32.465259077 (612h26m32.336342142s) [noise=true , address=145.79.227.34:4905 , piece_id=UXSGKRZ66C5I2JT4PAHWC5F35P34TFBJC623PMCUY5WE2W4OSEMA, seg_pos={0 0}, upload=536708]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9): protocol: expected piece hash; context canceled; EOF
Dec 31 03:55:32.465279145 (612h26m32.33636218s) [noise=true , address=145.79.227.34:4905 , piece_id=UXSGKRZ66C5I2JT4PAHWC5F35P34TFBJC623PMCUY5WE2W4OSEMA, seg_pos={0 0}, upload=536708]: Long tail cancellation due to optimal threshold
Dec 31 03:55:32.465312148 (612h26m32.336395193s) [seg_pos={0 0}, upload=536708]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 31 03:55:32.465413070 (612h26m32.336496115s) [upload=536708]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 31 03:55:32.465420875 (612h26m32.33650394s) [upload=536708]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 31 03:55:32.752465943 (612h26m32.623548979s) [upload=536708]: Done uploading segments...
Dec 31 03:55:32.752485711 (612h26m32.623568746s) [upload=536708]: Upload finished. err: <nil>
Duration: 6.71690589s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
23 out of 60 is failed (threshold=3) [...X.X.......X....X..X..X....X.XXXXX.X.....X.X...XXXX..XXXX.] duration:
5.758900073s trace:
https://jaeger.dev.storj.io/trace/6682252d7557c0f0
FAIL (2025-12-31 03:56:36 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: QT2ROG3IISNYE4QWIX7XFQH4CMTL6OT256X4IZP36O3U7DFJTNKQ) context deadline exceeded; piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: 4VON2DQN4YUE6OHKDMMIUIPM6VBA57F34UBAIEIAEGSDNU3AZEPA) context deadline exceeded; piecestore: (Node ID: 1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb, Piece ID: E4JIAWYCJF2QWHZRMMJCNDVWPG5SDJKF2RXXDVG7NMEEFTJZJXZA) context deadline exceeded
======LOGS======
Dec 31 03:55:29.610844489 (612h26m29.481927524s) [upload=536708]: Starting upload
Dec 31 03:55:29.610905506 (612h26m29.481988551s) [upload=536708]: Done starting upload
Dec 31 03:55:29.611054219 (612h26m29.482137254s) [upload=536708]: Uploading segments...
Dec 31 03:55:29.611720016 (612h26m29.482803061s) [upload=536708]: Got next segment. Inline: false
Dec 31 03:55:29.611774230 (612h26m29.482857265s) [upload=536708]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 31 03:55:29.611780622 (612h26m29.482863657s) [upload=536708]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 31 03:55:29.859696415 (612h26m29.73077945s) [seg_pos={0 0}, upload=536708]: Begin upload segment...
Dec 31 03:55:29.859753674 (612h26m29.730836719s) [seg_pos={0 0}, upload=536708]: Done begin upload segment.
Dec 31 03:55:29.859793790 (612h26m29.730876825s) [noise=true , address=77.90.31.21:4880 , piece_id=E4JIAWYCJF2QWHZRMMJCNDVWPG5SDJKF2RXXDVG7NMEEFTJZJXZA, seg_pos={0 0}, upload=536708]: Uploading piece...
Dec 31 03:55:29.859809850 (612h26m29.730892895s) [noise=true , address=145.79.227.19:4077 , piece_id=W4T3ATYZAI3YV6WD6UC3AXDGARHMC3HRXGXS25YBJSKLXD3JMV5Q, seg_pos={0 0}, upload=536708]: Uploading piece...
Dec 31 03:55:29.860369586 (612h26m29.731452621s) [noise=true , address=145.79.227.27:4690 , piece_id=5AWAMERXMY5LQH4HXEVTZ5JOPN5Z34HWPTSJKKDCDQYUFOXRIWDA, seg_pos={0 0}, upload=536708]: Uploading piece...
Dec 31 03:55:29.860847637 (612h26m29.731930672s) [noise=true , address=145.79.227.21:4474 , piece_id=TFDUOMBP474RIWGRHWLSL7TNOHNP7Q44Q6ADJKEB3DY54HQ5PRFQ, seg_pos={0 0}, upload=536708]: Uploading piece...
Dec 31 03:55:29.861240946 (612h26m29.732323981s) [noise=true , address=145.79.227.34:4905 , piece_id=UXSGKRZ66C5I2JT4PAHWC5F35P34TFBJC623PMCUY5WE2W4OSEMA, seg_pos={0 0}, upload=536708]: Uploading piece...
Dec 31 03:55:29.861247649 (612h26m29.732330694s) [noise=true , address=145.79.227.49:4337 , piece_id=VVUK7EQMF4XGVFMEBQBIPSTX5FBNMOAORY7DJUD4SMZCREBK7ANQ, seg_pos={0 0}, upload=536708]: Uploading piece...
Dec 31 03:55:29.861597725 (612h26m29.73268076s) [noise=true , address=77.90.31.11:4554 , piece_id=PKQ6I5WRTYV5MTOPMQON5ESRI4GT5KTFHSZZHHVYEYVLPW4BBG3Q, seg_pos={0 0}, upload=536708]: Uploading piece...
Dec 31 03:55:29.861846459 (612h26m29.732929494s) [noise=true , address=145.79.227.3:4149 , piece_id=JLV7CFVNSXDJFCKPCEE27UCSH5YVCTZJGOYFSQEDIO23IYM6QEOA, seg_pos={0 0}, upload=536708]: Uploading piece...
Dec 31 03:55:29.862245879 (612h26m29.733328914s) [noise=true , address=145.79.227.50:4375 , piece_id=5XEPQHLC2FKJIJGQXY7KH4B5ESOVUNEBTLFAQRQB6HJRWBVPHTDQ, seg_pos={0 0}, upload=536708]: Uploading piece...
Dec 31 03:55:29.862675798 (612h26m29.733758833s) [noise=true , address=77.90.31.10:4519 , piece_id=QT2ROG3IISNYE4QWIX7XFQH4CMTL6OT256X4IZP36O3U7DFJTNKQ, seg_pos={0 0}, upload=536708]: Uploading piece...
Dec 31 03:55:29.863048678 (612h26m29.734131713s) [noise=true , address=145.79.227.40:4049 , piece_id=3GVHPGMLAUUVRFEHO5BGP7UWCIQHCQ67RPK7TQV4BNEPFGG4ZSWA, seg_pos={0 0}, upload=536708]: Uploading piece...
Dec 31 03:55:29.863401139 (612h26m29.734484184s) [noise=true , address=145.79.227.52:4445 , piece_id=XDFC2E4K7P6ZP3T3QKFJOW6BNOJ5IGSRUX2W54HSG3F3FRGK5BUQ, seg_pos={0 0}, upload=536708]: Uploading piece...
Dec 31 03:55:29.863830968 (612h26m29.734914003s) [noise=true , address=45.91.116.9:4263 , piece_id=IQGT7QAUZUGTRRC7NL3IDUESBQANMOTGXN26KNBCXWBPUA6LFWBQ, seg_pos={0 0}, upload=536708]: Uploading piece...
Dec 31 03:55:29.864208145 (612h26m29.73529119s) [noise=true , address=145.79.227.42:4121 , piece_id=RZALVHACRGXFDA2TSN7M4KOSNMTVV6HF5VN56AVQBSLP57UDVYUA, seg_pos={0 0}, upload=536708]: Uploading piece...
Dec 31 03:55:29.864315971 (612h26m29.735399006s) [noise=true , address=45.91.116.13:4154 , piece_id=UMUMUAKI6DVU7E2URNXPX2HV7CA2QQMMFADPEF6KH6QKDNHPHNOA, seg_pos={0 0}, upload=536708]: Uploading piece...
Dec 31 03:55:29.864838185 (612h26m29.73592122s) [noise=true , address=45.91.116.6:4975 , piece_id=4VON2DQN4YUE6OHKDMMIUIPM6VBA57F34UBAIEIAEGSDNU3AZEPA, seg_pos={0 0}, upload=536708]: Uploading piece...
Dec 31 03:55:31.563613463 (612h26m31.434696498s) [upload=536708]: Next returned nil segment
Dec 31 03:55:31.563628091 (612h26m31.434711126s) [upload=536708]: Waiting for error group managing segments...
Dec 31 03:55:31.850844927 (612h26m31.721927972s) [noise=true , address=145.79.227.19:4077 , piece_id=W4T3ATYZAI3YV6WD6UC3AXDGARHMC3HRXGXS25YBJSKLXD3JMV5Q, seg_pos={0 0}, upload=536708]: Done uploading piece. err: <nil>
Dec 31 03:55:31.868260779 (612h26m31.739343825s) [noise=true , address=145.79.227.27:4690 , piece_id=5AWAMERXMY5LQH4HXEVTZ5JOPN5Z34HWPTSJKKDCDQYUFOXRIWDA, seg_pos={0 0}, upload=536708]: Done uploading piece. err: <nil>
Dec 31 03:55:31.872717676 (612h26m31.743800711s) [noise=true , address=145.79.227.52:4445 , piece_id=XDFC2E4K7P6ZP3T3QKFJOW6BNOJ5IGSRUX2W54HSG3F3FRGK5BUQ, seg_pos={0 0}, upload=536708]: Done uploading piece. err: <nil>
Dec 31 03:55:31.879399198 (612h26m31.750482243s) [noise=true , address=145.79.227.50:4375 , piece_id=5XEPQHLC2FKJIJGQXY7KH4B5ESOVUNEBTLFAQRQB6HJRWBVPHTDQ, seg_pos={0 0}, upload=536708]: Done uploading piece. err: <nil>
Dec 31 03:55:31.914183161 (612h26m31.785266196s) [noise=true , address=145.79.227.21:4474 , piece_id=TFDUOMBP474RIWGRHWLSL7TNOHNP7Q44Q6ADJKEB3DY54HQ5PRFQ, seg_pos={0 0}, upload=536708]: Done uploading piece. err: <nil>
Dec 31 03:55:31.932902375 (612h26m31.80398542s) [noise=true , address=145.79.227.42:4121 , piece_id=RZALVHACRGXFDA2TSN7M4KOSNMTVV6HF5VN56AVQBSLP57UDVYUA, seg_pos={0 0}, upload=536708]: Done uploading piece. err: <nil>
Dec 31 03:55:32.101017818 (612h26m31.972100853s) [noise=true , address=77.90.31.11:4554 , piece_id=PKQ6I5WRTYV5MTOPMQON5ESRI4GT5KTFHSZZHHVYEYVLPW4BBG3Q, seg_pos={0 0}, upload=536708]: Done uploading piece. err: <nil>
Dec 31 03:55:32.116944784 (612h26m31.988027829s) [noise=true , address=145.79.227.3:4149 , piece_id=JLV7CFVNSXDJFCKPCEE27UCSH5YVCTZJGOYFSQEDIO23IYM6QEOA, seg_pos={0 0}, upload=536708]: Done uploading piece. err: <nil>
Dec 31 03:55:32.135135732 (612h26m32.006218767s) [noise=true , address=77.90.31.21:4880 , piece_id=E4JIAWYCJF2QWHZRMMJCNDVWPG5SDJKF2RXXDVG7NMEEFTJZJXZA, seg_pos={0 0}, upload=536708]: Done uploading piece. err: <nil>
Dec 31 03:55:32.149722257 (612h26m32.020805292s) [noise=true , address=77.90.31.10:4519 , piece_id=QT2ROG3IISNYE4QWIX7XFQH4CMTL6OT256X4IZP36O3U7DFJTNKQ, seg_pos={0 0}, upload=536708]: Done uploading piece. err: <nil>
Dec 31 03:55:32.200260328 (612h26m32.071343373s) [noise=true , address=45.91.116.9:4263 , piece_id=IQGT7QAUZUGTRRC7NL3IDUESBQANMOTGXN26KNBCXWBPUA6LFWBQ, seg_pos={0 0}, upload=536708]: Done uploading piece. err: <nil>
Dec 31 03:55:32.224638697 (612h26m32.095721742s) [noise=true , address=45.91.116.6:4975 , piece_id=4VON2DQN4YUE6OHKDMMIUIPM6VBA57F34UBAIEIAEGSDNU3AZEPA, seg_pos={0 0}, upload=536708]: Done uploading piece. err: <nil>
Dec 31 03:55:32.464758784 (612h26m32.335841819s) [noise=true , address=45.91.116.13:4154 , piece_id=UMUMUAKI6DVU7E2URNXPX2HV7CA2QQMMFADPEF6KH6QKDNHPHNOA, seg_pos={0 0}, upload=536708]: Done uploading piece. err: <nil>
Dec 31 03:55:32.464778162 (612h26m32.335861197s) [seg_pos={0 0}, upload=536708]: Segment reached optimal threshold of 13 pieces.
Dec 31 03:55:32.464964857 (612h26m32.336047892s) [noise=true , address=145.79.227.40:4049 , piece_id=3GVHPGMLAUUVRFEHO5BGP7UWCIQHCQ67RPK7TQV4BNEPFGG4ZSWA, seg_pos={0 0}, upload=536708]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J): protocol: expected piece hash; context canceled; EOF
Dec 31 03:55:32.464974164 (612h26m32.336057199s) [noise=true , address=145.79.227.40:4049 , piece_id=3GVHPGMLAUUVRFEHO5BGP7UWCIQHCQ67RPK7TQV4BNEPFGG4ZSWA, seg_pos={0 0}, upload=536708]: Long tail cancellation due to optimal threshold
Dec 31 03:55:32.465155780 (612h26m32.336238815s) [noise=true , address=145.79.227.49:4337 , piece_id=VVUK7EQMF4XGVFMEBQBIPSTX5FBNMOAORY7DJUD4SMZCREBK7ANQ, seg_pos={0 0}, upload=536708]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nTZFfzNCcVAjut7NXf9seifysJDfvXkS8aZSnXKfw6gh1TrjE): protocol: expected piece hash; context canceled; EOF
Dec 31 03:55:32.465162413 (612h26m32.336245448s) [noise=true , address=145.79.227.49:4337 , piece_id=VVUK7EQMF4XGVFMEBQBIPSTX5FBNMOAORY7DJUD4SMZCREBK7ANQ, seg_pos={0 0}, upload=536708]: Long tail cancellation due to optimal threshold
Dec 31 03:55:32.465259077 (612h26m32.336342142s) [noise=true , address=145.79.227.34:4905 , piece_id=UXSGKRZ66C5I2JT4PAHWC5F35P34TFBJC623PMCUY5WE2W4OSEMA, seg_pos={0 0}, upload=536708]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9): protocol: expected piece hash; context canceled; EOF
Dec 31 03:55:32.465279145 (612h26m32.33636218s) [noise=true , address=145.79.227.34:4905 , piece_id=UXSGKRZ66C5I2JT4PAHWC5F35P34TFBJC623PMCUY5WE2W4OSEMA, seg_pos={0 0}, upload=536708]: Long tail cancellation due to optimal threshold
Dec 31 03:55:32.465312148 (612h26m32.336395193s) [seg_pos={0 0}, upload=536708]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 31 03:55:32.465413070 (612h26m32.336496115s) [upload=536708]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 31 03:55:32.465420875 (612h26m32.33650394s) [upload=536708]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 31 03:55:32.752465943 (612h26m32.623548979s) [upload=536708]: Done uploading segments...
Dec 31 03:55:32.752485711 (612h26m32.623568746s) [upload=536708]: Upload finished. err: <nil>
Duration: 5.758964055s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
16 out of 60 is failed (threshold=6) [X...X...X...X.X.X...X......X..XX.....X....X............XX.XX] duration:
5.001939328s trace:
https://jaeger.dev.storj.io/trace/1bf049201ba7735a
FAIL (2025-12-31 03:56:33 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001959136s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [..................X.........................................] duration:
49.563415432s trace:
https://jaeger.dev.storj.io/trace/634c422994952cfc
FAIL (2025-12-31 03:56:28 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 121Xa3rvgKQhHc5NBufs57Ybnw3fYKUjgRyK32TTwWoi4E8m57o, Piece ID: 7ZJODSC7K4XWDIBJGMYHMGRFCXZGHTSGWTWPO7NKPWGGT73NFGOQ) context deadline exceeded; piecestore: (Node ID: 12caEiJobigXcGWwaX3Vt16zd2NXqk1WzLWxZME44E8Q8wdhn3q, Piece ID: RTZZG5ZDBPM74ZXITWZZ6LKJCKUOQ3YY4UUTMZ5VRVKFZE4YKAUA) context deadline exceeded; piecestore: (Node ID: 1zwuzCLWy2q166Ld52WUG9fvXNdJWsQrg2NearHYg7pyxpXFen, Piece ID: EJWBT3NCXTW5JSOIICAASNNDWFMIM2AO2SAE26S5POMMLEOYUH7Q) context deadline exceeded
======LOGS======
) [noise=true , address=193.149.164.143:4704 , piece_id=XIHZWOG63PEC3AQOHHWBLF53Y3B42NOO6TRNEUR3CIJB3YZTX3LQ, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.363237580 (611h51m51.234320615s) [noise=true , address=145.79.227.15:4889 , piece_id=EACS3IK7KTZI73UJEUOZB33TZN64WDWRC7TV5YJZ5ZMQVATQ3ORQ, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.364121825 (611h51m51.23520486s) [noise=true , address=193.149.164.143:4699 , piece_id=5LCUJBU4MEYEA7CQCA2FVVX6VCEX7SU3W4N3BBPQMLP6AHG5ZKKQ, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.364545251 (611h51m51.235628286s) [noise=true , address=45.91.116.11:4143 , piece_id=RTZZG5ZDBPM74ZXITWZZ6LKJCKUOQ3YY4UUTMZ5VRVKFZE4YKAUA, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.364959129 (611h51m51.236042174s) [noise=true , address=145.79.227.3:4176 , piece_id=TBPK43ULJ75RKDV5P63GXNVYZWG7TIQ3YLKMBLU55OQ2PFGSBH7A, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.365381323 (611h51m51.236464368s) [noise=true , address=145.79.227.44:4217 , piece_id=B5JSD2DUWSMSEME5CZCWFBXELK4WRMAXLLA2QQL73SB6WR6ECXUA, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.367027327 (611h51m51.238110362s) [noise=true , address=77.90.31.11:4561 , piece_id=IRE2JNK63DRH52RXBC5D3MRL6QO5S5N44UFVFWDRNY7LZBQ3IFMQ, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.367801231 (611h51m51.238884326s) [noise=true , address=145.79.227.40:4058 , piece_id=ECUZDPLHYQW3KYSTHKWBPD3CCTUCMAZ6XMOSH5S46ACF2T5366BQ, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.369086348 (611h51m51.240169404s) [noise=true , address=77.90.31.10:4536 , piece_id=EJWBT3NCXTW5JSOIICAASNNDWFMIM2AO2SAE26S5POMMLEOYUH7Q, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.370451017 (611h51m51.241534062s) [noise=true , address=77.90.31.19:4861 , piece_id=U7I2SDPMKDL4RLP2CIWZUSCEWPM5LP67VMUOLFKBHHVSCXL55MYQ, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.371980618 (611h51m51.243063654s) [noise=true , address=64.49.13.26:4955 , piece_id=53IGVDHKGGBIEKVJ6A52Q647E74KVI4FFRQU7VYK7POFU4CCX4HA, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.374009812 (611h51m51.245092847s) [noise=true , address=77.90.31.14:4671 , piece_id=2YXJ5RJERCNTD3TJHJHF4WBHR3DEOHDIOWI53H4PDS4EWQECEU2A, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.374500948 (611h51m51.245583993s) [noise=true , address=77.90.31.13:4632 , piece_id=72ADTLC2JC7Z3Z4GYKPECNAK6QMCMONGLUKZKQFLWN5UIQ4IPZYQ, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.392662589 (611h51m51.263745634s) [noise=true , address=145.79.227.32:4865 , piece_id=ZXAGYGEC35VVET2IAX2KJII4PUK3DFQXOFAYCC5R7KV5PNQCSUIA, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.393651513 (611h51m51.264734558s) [noise=true , address=145.79.227.31:4804 , piece_id=4E3PGEBPB2BFUAHNXJI4MZFWNNNZMMFDF5UBTT2UM4F2NHOHCIQQ, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.396266312 (611h51m51.267349357s) [noise=true , address=145.79.227.29:4966 , piece_id=N4JGQH23KY2BMVZZWWX7CKZN2FC2FJBFNAZEGZ7CRMRQAYEAZJLA, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:52.828565815 (611h51m52.69964886s) [upload=536207]: Next returned nil segment
Dec 31 03:20:52.828580594 (611h51m52.699663629s) [upload=536207]: Waiting for error group managing segments...
Dec 31 03:20:53.204430167 (611h51m53.075513202s) [noise=true , address=145.79.227.15:4889 , piece_id=EACS3IK7KTZI73UJEUOZB33TZN64WDWRC7TV5YJZ5ZMQVATQ3ORQ, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:53.265960889 (611h51m53.137043924s) [noise=true , address=145.79.227.31:4804 , piece_id=4E3PGEBPB2BFUAHNXJI4MZFWNNNZMMFDF5UBTT2UM4F2NHOHCIQQ, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:53.266193351 (611h51m53.137276386s) [noise=true , address=145.79.227.32:4865 , piece_id=ZXAGYGEC35VVET2IAX2KJII4PUK3DFQXOFAYCC5R7KV5PNQCSUIA, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:53.331763405 (611h51m53.20284644s) [noise=true , address=5.231.73.2:4035 , piece_id=7ZJODSC7K4XWDIBJGMYHMGRFCXZGHTSGWTWPO7NKPWGGT73NFGOQ, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:53.412602422 (611h51m53.283685467s) [noise=true , address=77.90.31.10:4536 , piece_id=EJWBT3NCXTW5JSOIICAASNNDWFMIM2AO2SAE26S5POMMLEOYUH7Q, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:53.419806240 (611h51m53.290889275s) [noise=true , address=77.90.31.14:4671 , piece_id=2YXJ5RJERCNTD3TJHJHF4WBHR3DEOHDIOWI53H4PDS4EWQECEU2A, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:53.498590626 (611h51m53.369673671s) [noise=true , address=77.90.31.11:4561 , piece_id=IRE2JNK63DRH52RXBC5D3MRL6QO5S5N44UFVFWDRNY7LZBQ3IFMQ, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:53.520983294 (611h51m53.392066339s) [noise=true , address=45.91.116.11:4143 , piece_id=RTZZG5ZDBPM74ZXITWZZ6LKJCKUOQ3YY4UUTMZ5VRVKFZE4YKAUA, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:53.585203506 (611h51m53.456286541s) [noise=true , address=77.90.31.19:4861 , piece_id=U7I2SDPMKDL4RLP2CIWZUSCEWPM5LP67VMUOLFKBHHVSCXL55MYQ, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:54.188950073 (611h51m54.060033119s) [noise=true , address=145.79.227.44:4217 , piece_id=B5JSD2DUWSMSEME5CZCWFBXELK4WRMAXLLA2QQL73SB6WR6ECXUA, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:54.204771008 (611h51m54.075854053s) [noise=true , address=145.79.227.40:4058 , piece_id=ECUZDPLHYQW3KYSTHKWBPD3CCTUCMAZ6XMOSH5S46ACF2T5366BQ, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:54.231988834 (611h51m54.103071879s) [noise=true , address=193.149.164.143:4699 , piece_id=5LCUJBU4MEYEA7CQCA2FVVX6VCEX7SU3W4N3BBPQMLP6AHG5ZKKQ, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:54.275280076 (611h51m54.146363121s) [noise=true , address=145.79.227.29:4966 , piece_id=N4JGQH23KY2BMVZZWWX7CKZN2FC2FJBFNAZEGZ7CRMRQAYEAZJLA, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:54.307143628 (611h51m54.178226673s) [noise=true , address=77.90.31.12:4602 , piece_id=ZPVIMHS5CQAKREW456SPNTHR3WM6IWZDB6AJZIYFREALJ6V63O6A, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:54.307168195 (611h51m54.17825124s) [seg_pos={0 0}, upload=536207]: Segment reached optimal threshold of 14 pieces.
Dec 31 03:20:54.307445734 (611h51m54.178528789s) [noise=true , address=77.90.31.13:4632 , piece_id=72ADTLC2JC7Z3Z4GYKPECNAK6QMCMONGLUKZKQFLWN5UIQ4IPZYQ, seg_pos={0 0}, upload=536207]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13iPed9zYrthvh36xo3k4gsqm4tz86iVkNvAr3wauuArQLquyS): protocol: expected piece hash; context canceled; EOF
Dec 31 03:20:54.307457145 (611h51m54.1785402s) [noise=true , address=77.90.31.13:4632 , piece_id=72ADTLC2JC7Z3Z4GYKPECNAK6QMCMONGLUKZKQFLWN5UIQ4IPZYQ, seg_pos={0 0}, upload=536207]: Long tail cancellation due to optimal threshold
Dec 31 03:20:54.307494436 (611h51m54.178577471s) [noise=true , address=193.149.164.143:4704 , piece_id=XIHZWOG63PEC3AQOHHWBLF53Y3B42NOO6TRNEUR3CIJB3YZTX3LQ, seg_pos={0 0}, upload=536207]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bUVmWsfAfQiPWhDkYRNGhims8TLQqGqvhrZMYugLLTdmLk2xY): protocol: expected piece hash; context canceled; EOF
Dec 31 03:20:54.307513032 (611h51m54.178596067s) [noise=true , address=193.149.164.143:4704 , piece_id=XIHZWOG63PEC3AQOHHWBLF53Y3B42NOO6TRNEUR3CIJB3YZTX3LQ, seg_pos={0 0}, upload=536207]: Long tail cancellation due to optimal threshold
Dec 31 03:20:54.307792985 (611h51m54.17887602s) [noise=true , address=145.79.227.3:4176 , piece_id=TBPK43ULJ75RKDV5P63GXNVYZWG7TIQ3YLKMBLU55OQ2PFGSBH7A, seg_pos={0 0}, upload=536207]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kGgrsNn5tv2GkcuVHHArCACwRB2qwbj3X3JmzVF4UL6dLRW1m): protocol: write tcp 188.166.225.170:39612->145.79.227.3:4176: use of closed network connection; write tcp 188.166.225.170:39612->145.79.227.3:4176: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39612->145.79.227.3:4176: use of closed network connection
Dec 31 03:20:54.307803214 (611h51m54.178886259s) [noise=true , address=145.79.227.3:4176 , piece_id=TBPK43ULJ75RKDV5P63GXNVYZWG7TIQ3YLKMBLU55OQ2PFGSBH7A, seg_pos={0 0}, upload=536207]: Long tail cancellation due to optimal threshold
Dec 31 03:20:54.307870673 (611h51m54.178953718s) [noise=true , address=64.49.13.26:4955 , piece_id=53IGVDHKGGBIEKVJ6A52Q647E74KVI4FFRQU7VYK7POFU4CCX4HA, seg_pos={0 0}, upload=536207]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125TV3v1CzV1Q9TPfyVHsn26NZ5XcHixKxA66t96iz62FF4SHAP): protocol: expected piece hash; context canceled; EOF
Dec 31 03:20:54.307882294 (611h51m54.178965339s) [noise=true , address=64.49.13.26:4955 , piece_id=53IGVDHKGGBIEKVJ6A52Q647E74KVI4FFRQU7VYK7POFU4CCX4HA, seg_pos={0 0}, upload=536207]: Long tail cancellation due to optimal threshold
Dec 31 03:20:54.307909116 (611h51m54.178992151s) [seg_pos={0 0}, upload=536207]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 31 03:20:54.308005189 (611h51m54.179088234s) [upload=536207]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 31 03:20:54.308013014 (611h51m54.179096049s) [upload=536207]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 31 03:20:54.593642638 (611h51m54.464725683s) [upload=536207]: Done uploading segments...
Dec 31 03:20:54.593663947 (611h51m54.464746983s) [upload=536207]: Upload finished. err: <nil>
Duration: 49.563436913s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [................X....X.X..X.................................] duration:
15.79367532s trace:
https://jaeger.dev.storj.io/trace/5e7c34cde28aa313
FAIL (2025-12-31 03:56:06 UTC)
orbiter warning: context deadline exceeded
Duration: 15.793705728s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
24 out of 60 is failed (threshold=3) [X...X.X.......X....X..X..X....X.XXXXX.X.....X.X...XXXX..XXXX] duration:
30.002731785s trace:
https://jaeger.dev.storj.io/trace/6682252d7557c0f0
FAIL (2025-12-31 03:55:58 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: QT2ROG3IISNYE4QWIX7XFQH4CMTL6OT256X4IZP36O3U7DFJTNKQ) context deadline exceeded; piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: 4VON2DQN4YUE6OHKDMMIUIPM6VBA57F34UBAIEIAEGSDNU3AZEPA) context deadline exceeded; piecestore: (Node ID: 1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb, Piece ID: E4JIAWYCJF2QWHZRMMJCNDVWPG5SDJKF2RXXDVG7NMEEFTJZJXZA) context deadline exceeded
======LOGS======
Dec 31 03:55:29.610844489 (612h26m29.481927524s) [upload=536708]: Starting upload
Dec 31 03:55:29.610905506 (612h26m29.481988551s) [upload=536708]: Done starting upload
Dec 31 03:55:29.611054219 (612h26m29.482137254s) [upload=536708]: Uploading segments...
Dec 31 03:55:29.611720016 (612h26m29.482803061s) [upload=536708]: Got next segment. Inline: false
Dec 31 03:55:29.611774230 (612h26m29.482857265s) [upload=536708]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 31 03:55:29.611780622 (612h26m29.482863657s) [upload=536708]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 31 03:55:29.859696415 (612h26m29.73077945s) [seg_pos={0 0}, upload=536708]: Begin upload segment...
Dec 31 03:55:29.859753674 (612h26m29.730836719s) [seg_pos={0 0}, upload=536708]: Done begin upload segment.
Dec 31 03:55:29.859793790 (612h26m29.730876825s) [noise=true , address=77.90.31.21:4880 , piece_id=E4JIAWYCJF2QWHZRMMJCNDVWPG5SDJKF2RXXDVG7NMEEFTJZJXZA, seg_pos={0 0}, upload=536708]: Uploading piece...
Dec 31 03:55:29.859809850 (612h26m29.730892895s) [noise=true , address=145.79.227.19:4077 , piece_id=W4T3ATYZAI3YV6WD6UC3AXDGARHMC3HRXGXS25YBJSKLXD3JMV5Q, seg_pos={0 0}, upload=536708]: Uploading piece...
Dec 31 03:55:29.860369586 (612h26m29.731452621s) [noise=true , address=145.79.227.27:4690 , piece_id=5AWAMERXMY5LQH4HXEVTZ5JOPN5Z34HWPTSJKKDCDQYUFOXRIWDA, seg_pos={0 0}, upload=536708]: Uploading piece...
Dec 31 03:55:29.860847637 (612h26m29.731930672s) [noise=true , address=145.79.227.21:4474 , piece_id=TFDUOMBP474RIWGRHWLSL7TNOHNP7Q44Q6ADJKEB3DY54HQ5PRFQ, seg_pos={0 0}, upload=536708]: Uploading piece...
Dec 31 03:55:29.861240946 (612h26m29.732323981s) [noise=true , address=145.79.227.34:4905 , piece_id=UXSGKRZ66C5I2JT4PAHWC5F35P34TFBJC623PMCUY5WE2W4OSEMA, seg_pos={0 0}, upload=536708]: Uploading piece...
Dec 31 03:55:29.861247649 (612h26m29.732330694s) [noise=true , address=145.79.227.49:4337 , piece_id=VVUK7EQMF4XGVFMEBQBIPSTX5FBNMOAORY7DJUD4SMZCREBK7ANQ, seg_pos={0 0}, upload=536708]: Uploading piece...
Dec 31 03:55:29.861597725 (612h26m29.73268076s) [noise=true , address=77.90.31.11:4554 , piece_id=PKQ6I5WRTYV5MTOPMQON5ESRI4GT5KTFHSZZHHVYEYVLPW4BBG3Q, seg_pos={0 0}, upload=536708]: Uploading piece...
Dec 31 03:55:29.861846459 (612h26m29.732929494s) [noise=true , address=145.79.227.3:4149 , piece_id=JLV7CFVNSXDJFCKPCEE27UCSH5YVCTZJGOYFSQEDIO23IYM6QEOA, seg_pos={0 0}, upload=536708]: Uploading piece...
Dec 31 03:55:29.862245879 (612h26m29.733328914s) [noise=true , address=145.79.227.50:4375 , piece_id=5XEPQHLC2FKJIJGQXY7KH4B5ESOVUNEBTLFAQRQB6HJRWBVPHTDQ, seg_pos={0 0}, upload=536708]: Uploading piece...
Dec 31 03:55:29.862675798 (612h26m29.733758833s) [noise=true , address=77.90.31.10:4519 , piece_id=QT2ROG3IISNYE4QWIX7XFQH4CMTL6OT256X4IZP36O3U7DFJTNKQ, seg_pos={0 0}, upload=536708]: Uploading piece...
Dec 31 03:55:29.863048678 (612h26m29.734131713s) [noise=true , address=145.79.227.40:4049 , piece_id=3GVHPGMLAUUVRFEHO5BGP7UWCIQHCQ67RPK7TQV4BNEPFGG4ZSWA, seg_pos={0 0}, upload=536708]: Uploading piece...
Dec 31 03:55:29.863401139 (612h26m29.734484184s) [noise=true , address=145.79.227.52:4445 , piece_id=XDFC2E4K7P6ZP3T3QKFJOW6BNOJ5IGSRUX2W54HSG3F3FRGK5BUQ, seg_pos={0 0}, upload=536708]: Uploading piece...
Dec 31 03:55:29.863830968 (612h26m29.734914003s) [noise=true , address=45.91.116.9:4263 , piece_id=IQGT7QAUZUGTRRC7NL3IDUESBQANMOTGXN26KNBCXWBPUA6LFWBQ, seg_pos={0 0}, upload=536708]: Uploading piece...
Dec 31 03:55:29.864208145 (612h26m29.73529119s) [noise=true , address=145.79.227.42:4121 , piece_id=RZALVHACRGXFDA2TSN7M4KOSNMTVV6HF5VN56AVQBSLP57UDVYUA, seg_pos={0 0}, upload=536708]: Uploading piece...
Dec 31 03:55:29.864315971 (612h26m29.735399006s) [noise=true , address=45.91.116.13:4154 , piece_id=UMUMUAKI6DVU7E2URNXPX2HV7CA2QQMMFADPEF6KH6QKDNHPHNOA, seg_pos={0 0}, upload=536708]: Uploading piece...
Dec 31 03:55:29.864838185 (612h26m29.73592122s) [noise=true , address=45.91.116.6:4975 , piece_id=4VON2DQN4YUE6OHKDMMIUIPM6VBA57F34UBAIEIAEGSDNU3AZEPA, seg_pos={0 0}, upload=536708]: Uploading piece...
Dec 31 03:55:31.563613463 (612h26m31.434696498s) [upload=536708]: Next returned nil segment
Dec 31 03:55:31.563628091 (612h26m31.434711126s) [upload=536708]: Waiting for error group managing segments...
Dec 31 03:55:31.850844927 (612h26m31.721927972s) [noise=true , address=145.79.227.19:4077 , piece_id=W4T3ATYZAI3YV6WD6UC3AXDGARHMC3HRXGXS25YBJSKLXD3JMV5Q, seg_pos={0 0}, upload=536708]: Done uploading piece. err: <nil>
Dec 31 03:55:31.868260779 (612h26m31.739343825s) [noise=true , address=145.79.227.27:4690 , piece_id=5AWAMERXMY5LQH4HXEVTZ5JOPN5Z34HWPTSJKKDCDQYUFOXRIWDA, seg_pos={0 0}, upload=536708]: Done uploading piece. err: <nil>
Dec 31 03:55:31.872717676 (612h26m31.743800711s) [noise=true , address=145.79.227.52:4445 , piece_id=XDFC2E4K7P6ZP3T3QKFJOW6BNOJ5IGSRUX2W54HSG3F3FRGK5BUQ, seg_pos={0 0}, upload=536708]: Done uploading piece. err: <nil>
Dec 31 03:55:31.879399198 (612h26m31.750482243s) [noise=true , address=145.79.227.50:4375 , piece_id=5XEPQHLC2FKJIJGQXY7KH4B5ESOVUNEBTLFAQRQB6HJRWBVPHTDQ, seg_pos={0 0}, upload=536708]: Done uploading piece. err: <nil>
Dec 31 03:55:31.914183161 (612h26m31.785266196s) [noise=true , address=145.79.227.21:4474 , piece_id=TFDUOMBP474RIWGRHWLSL7TNOHNP7Q44Q6ADJKEB3DY54HQ5PRFQ, seg_pos={0 0}, upload=536708]: Done uploading piece. err: <nil>
Dec 31 03:55:31.932902375 (612h26m31.80398542s) [noise=true , address=145.79.227.42:4121 , piece_id=RZALVHACRGXFDA2TSN7M4KOSNMTVV6HF5VN56AVQBSLP57UDVYUA, seg_pos={0 0}, upload=536708]: Done uploading piece. err: <nil>
Dec 31 03:55:32.101017818 (612h26m31.972100853s) [noise=true , address=77.90.31.11:4554 , piece_id=PKQ6I5WRTYV5MTOPMQON5ESRI4GT5KTFHSZZHHVYEYVLPW4BBG3Q, seg_pos={0 0}, upload=536708]: Done uploading piece. err: <nil>
Dec 31 03:55:32.116944784 (612h26m31.988027829s) [noise=true , address=145.79.227.3:4149 , piece_id=JLV7CFVNSXDJFCKPCEE27UCSH5YVCTZJGOYFSQEDIO23IYM6QEOA, seg_pos={0 0}, upload=536708]: Done uploading piece. err: <nil>
Dec 31 03:55:32.135135732 (612h26m32.006218767s) [noise=true , address=77.90.31.21:4880 , piece_id=E4JIAWYCJF2QWHZRMMJCNDVWPG5SDJKF2RXXDVG7NMEEFTJZJXZA, seg_pos={0 0}, upload=536708]: Done uploading piece. err: <nil>
Dec 31 03:55:32.149722257 (612h26m32.020805292s) [noise=true , address=77.90.31.10:4519 , piece_id=QT2ROG3IISNYE4QWIX7XFQH4CMTL6OT256X4IZP36O3U7DFJTNKQ, seg_pos={0 0}, upload=536708]: Done uploading piece. err: <nil>
Dec 31 03:55:32.200260328 (612h26m32.071343373s) [noise=true , address=45.91.116.9:4263 , piece_id=IQGT7QAUZUGTRRC7NL3IDUESBQANMOTGXN26KNBCXWBPUA6LFWBQ, seg_pos={0 0}, upload=536708]: Done uploading piece. err: <nil>
Dec 31 03:55:32.224638697 (612h26m32.095721742s) [noise=true , address=45.91.116.6:4975 , piece_id=4VON2DQN4YUE6OHKDMMIUIPM6VBA57F34UBAIEIAEGSDNU3AZEPA, seg_pos={0 0}, upload=536708]: Done uploading piece. err: <nil>
Dec 31 03:55:32.464758784 (612h26m32.335841819s) [noise=true , address=45.91.116.13:4154 , piece_id=UMUMUAKI6DVU7E2URNXPX2HV7CA2QQMMFADPEF6KH6QKDNHPHNOA, seg_pos={0 0}, upload=536708]: Done uploading piece. err: <nil>
Dec 31 03:55:32.464778162 (612h26m32.335861197s) [seg_pos={0 0}, upload=536708]: Segment reached optimal threshold of 13 pieces.
Dec 31 03:55:32.464964857 (612h26m32.336047892s) [noise=true , address=145.79.227.40:4049 , piece_id=3GVHPGMLAUUVRFEHO5BGP7UWCIQHCQ67RPK7TQV4BNEPFGG4ZSWA, seg_pos={0 0}, upload=536708]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J): protocol: expected piece hash; context canceled; EOF
Dec 31 03:55:32.464974164 (612h26m32.336057199s) [noise=true , address=145.79.227.40:4049 , piece_id=3GVHPGMLAUUVRFEHO5BGP7UWCIQHCQ67RPK7TQV4BNEPFGG4ZSWA, seg_pos={0 0}, upload=536708]: Long tail cancellation due to optimal threshold
Dec 31 03:55:32.465155780 (612h26m32.336238815s) [noise=true , address=145.79.227.49:4337 , piece_id=VVUK7EQMF4XGVFMEBQBIPSTX5FBNMOAORY7DJUD4SMZCREBK7ANQ, seg_pos={0 0}, upload=536708]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nTZFfzNCcVAjut7NXf9seifysJDfvXkS8aZSnXKfw6gh1TrjE): protocol: expected piece hash; context canceled; EOF
Dec 31 03:55:32.465162413 (612h26m32.336245448s) [noise=true , address=145.79.227.49:4337 , piece_id=VVUK7EQMF4XGVFMEBQBIPSTX5FBNMOAORY7DJUD4SMZCREBK7ANQ, seg_pos={0 0}, upload=536708]: Long tail cancellation due to optimal threshold
Dec 31 03:55:32.465259077 (612h26m32.336342142s) [noise=true , address=145.79.227.34:4905 , piece_id=UXSGKRZ66C5I2JT4PAHWC5F35P34TFBJC623PMCUY5WE2W4OSEMA, seg_pos={0 0}, upload=536708]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9): protocol: expected piece hash; context canceled; EOF
Dec 31 03:55:32.465279145 (612h26m32.33636218s) [noise=true , address=145.79.227.34:4905 , piece_id=UXSGKRZ66C5I2JT4PAHWC5F35P34TFBJC623PMCUY5WE2W4OSEMA, seg_pos={0 0}, upload=536708]: Long tail cancellation due to optimal threshold
Dec 31 03:55:32.465312148 (612h26m32.336395193s) [seg_pos={0 0}, upload=536708]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 31 03:55:32.465413070 (612h26m32.336496115s) [upload=536708]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 31 03:55:32.465420875 (612h26m32.33650394s) [upload=536708]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 31 03:55:32.752465943 (612h26m32.623548979s) [upload=536708]: Done uploading segments...
Dec 31 03:55:32.752485711 (612h26m32.623568746s) [upload=536708]: Upload finished. err: <nil>
Duration: 30.002754018s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
15 out of 60 is failed (threshold=6) [.X...X...X...X.X.X...X......X..XX.....X....X............XX.X] duration:
5.000867773s trace:
https://jaeger.dev.storj.io/trace/3f5ad6e59ecedf64
FAIL (2025-12-31 03:55:42 UTC)
orbiter warning: context deadline exceeded
Duration: 5.000892671s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
24 out of 60 is failed (threshold=3) [XX...X.X.......X....X..X..X....X.XXXXX.X.....X.X...XXXX..XXX] duration:
30.003086149s trace:
https://jaeger.dev.storj.io/trace/393fa5bd17dc0d30
FAIL (2025-12-31 03:55:27 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, Piece ID: IXDW2KUODOF2LYPJELC2RIUS7D6RLKNTX6QOHRGZ5HWBHX7D2Q5A) context deadline exceeded; piecestore: (Node ID: 121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp, Piece ID: SUSG3GFUSNXCJPYMPEXECHN4D7V7KJXB4CVNIGTF5CF5E2NT72OA) context deadline exceeded
======LOGS======
Dec 31 03:54:58.602751213 (612h25m58.473834248s) [upload=536701]: Starting upload
Dec 31 03:54:58.602808252 (612h25m58.473891287s) [upload=536701]: Done starting upload
Dec 31 03:54:58.602905197 (612h25m58.473988242s) [upload=536701]: Uploading segments...
Dec 31 03:54:58.603073828 (612h25m58.474156873s) [upload=536701]: Got next segment. Inline: false
Dec 31 03:54:58.603579160 (612h25m58.474662195s) [upload=536701]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 31 03:54:58.603587365 (612h25m58.47467041s) [upload=536701]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 31 03:54:58.850092642 (612h25m58.721175687s) [seg_pos={0 0}, upload=536701]: Begin upload segment...
Dec 31 03:54:58.850162475 (612h25m58.72124552s) [seg_pos={0 0}, upload=536701]: Done begin upload segment.
Dec 31 03:54:58.850218653 (612h25m58.721301688s) [noise=true , address=145.79.227.24:4582 , piece_id=R3VT77IS4WJREUOCUECGEG2CFML3QFH6GECZILMIGD2VG5TPNI4A, seg_pos={0 0}, upload=536701]: Uploading piece...
Dec 31 03:54:58.850245142 (612h25m58.721328177s) [noise=true , address=145.79.227.34:4905 , piece_id=O4HW7RRUM4TTXT5XXLW3MLWGETACA2VRMF5E7N2ESPHJOHEPPZ4A, seg_pos={0 0}, upload=536701]: Uploading piece...
Dec 31 03:54:58.851133134 (612h25m58.722216169s) [noise=true , address=145.79.227.22:4511 , piece_id=BGTSHILL2Z5FSLRHZW5IOU45NW33BP7MH6LL5LAOBXQI62SUSH3Q, seg_pos={0 0}, upload=536701]: Uploading piece...
Dec 31 03:54:58.851444437 (612h25m58.722527482s) [noise=true , address=145.79.227.19:4077 , piece_id=UHZ2WLGPXVUR34YKFAV74WCU5R4MVXNM54GAPALX2W4BYCG7GRCQ, seg_pos={0 0}, upload=536701]: Uploading piece...
Dec 31 03:54:58.851508338 (612h25m58.722591374s) [noise=true , address=145.79.227.12:4767 , piece_id=5B4HV5JDM7BANTF7V6JHOG45M6ZEG5JQHW3EACHKW63QQDX5FGCA, seg_pos={0 0}, upload=536701]: Uploading piece...
Dec 31 03:54:58.851871259 (612h25m58.722954294s) [noise=true , address=145.79.227.48:4303 , piece_id=2ZBODVLWTZ4YFF6NLH6BHVLUR5MINB6SXJNVXGC5SLJQTZNA7SSQ, seg_pos={0 0}, upload=536701]: Uploading piece...
Dec 31 03:54:58.851953136 (612h25m58.723036191s) [noise=true , address=145.79.227.50:4374 , piece_id=7NK5CN3C5TWMS4PAKPGVQMJTWUIBSOPC62ED4CBRHUUXABCYYSFQ, seg_pos={0 0}, upload=536701]: Uploading piece...
Dec 31 03:54:58.852194785 (612h25m58.72327782s) [noise=true , address=145.79.227.26:4653 , piece_id=QXT7VXK2OMJTVEYESIJWF47S3UWOP6G5PHKDGR5HB2Q6T2CVXMPQ, seg_pos={0 0}, upload=536701]: Uploading piece...
Dec 31 03:54:58.852435974 (612h25m58.723519009s) [noise=true , address=145.79.227.17:4971 , piece_id=4WLASATBOUBC4WCWIGGEP7CBC6EE2GM7SPD7BLRRSEXII5JULG6Q, seg_pos={0 0}, upload=536701]: Uploading piece...
Dec 31 03:54:58.852690449 (612h25m58.723773495s) [noise=true , address=145.79.227.13:4802 , piece_id=YT52KDM7B72ZP4RDGUUNK3XFTR7U4PQX27ZPQP3OUG2WDRQHSVOA, seg_pos={0 0}, upload=536701]: Uploading piece...
Dec 31 03:54:58.852867126 (612h25m58.723950161s) [noise=true , address=145.79.227.38:4977 , piece_id=OJCBLWCMHRLDZNZEHIVEN4PMX424XOFN55DWOLVBQ3EDIEWYOJSA, seg_pos={0 0}, upload=536701]: Uploading piece...
Dec 31 03:54:58.853046217 (612h25m58.724129252s) [noise=true , address=45.91.116.6:4976 , piece_id=SUSG3GFUSNXCJPYMPEXECHN4D7V7KJXB4CVNIGTF5CF5E2NT72OA, seg_pos={0 0}, upload=536701]: Uploading piece...
Dec 31 03:54:58.853337321 (612h25m58.724420356s) [noise=true , address=45.91.116.11:4119 , piece_id=IXDW2KUODOF2LYPJELC2RIUS7D6RLKNTX6QOHRGZ5HWBHX7D2Q5A, seg_pos={0 0}, upload=536701]: Uploading piece...
Dec 31 03:54:58.853524708 (612h25m58.724607743s) [noise=true , address=145.79.227.23:4546 , piece_id=DMYC6JLPVLF5JO4MDNSWYLC5HECZ4ICSEQXIQACCIAOHO5LCSKKQ, seg_pos={0 0}, upload=536701]: Uploading piece...
Dec 31 03:54:58.853994682 (612h25m58.725077728s) [noise=true , address=77.90.31.15:4700 , piece_id=43G6PMKRQBILI5ZGYW5QJN2SEMVZDXORRGKJOT5MCDZVO22DRE6Q, seg_pos={0 0}, upload=536701]: Uploading piece...
Dec 31 03:54:58.854535563 (612h25m58.725618598s) [noise=true , address=145.79.227.30:4762 , piece_id=D73FMYOOPAVYNACGUPAUEF6ID7ZZBQLO4QP2URVDBNDVRHCKL62A, seg_pos={0 0}, upload=536701]: Uploading piece...
Dec 31 03:55:00.543277716 (612h26m0.414360752s) [upload=536701]: Next returned nil segment
Dec 31 03:55:00.543295230 (612h26m0.414378285s) [upload=536701]: Waiting for error group managing segments...
Dec 31 03:55:00.835195434 (612h26m0.706278469s) [noise=true , address=145.79.227.48:4303 , piece_id=2ZBODVLWTZ4YFF6NLH6BHVLUR5MINB6SXJNVXGC5SLJQTZNA7SSQ, seg_pos={0 0}, upload=536701]: Done uploading piece. err: <nil>
Dec 31 03:55:00.837045467 (612h26m0.708128512s) [noise=true , address=145.79.227.19:4077 , piece_id=UHZ2WLGPXVUR34YKFAV74WCU5R4MVXNM54GAPALX2W4BYCG7GRCQ, seg_pos={0 0}, upload=536701]: Done uploading piece. err: <nil>
Dec 31 03:55:00.839535147 (612h26m0.710618192s) [noise=true , address=145.79.227.17:4971 , piece_id=4WLASATBOUBC4WCWIGGEP7CBC6EE2GM7SPD7BLRRSEXII5JULG6Q, seg_pos={0 0}, upload=536701]: Done uploading piece. err: <nil>
Dec 31 03:55:00.844570045 (612h26m0.71565308s) [noise=true , address=145.79.227.23:4546 , piece_id=DMYC6JLPVLF5JO4MDNSWYLC5HECZ4ICSEQXIQACCIAOHO5LCSKKQ, seg_pos={0 0}, upload=536701]: Done uploading piece. err: <nil>
Dec 31 03:55:00.851457910 (612h26m0.722540945s) [noise=true , address=145.79.227.24:4582 , piece_id=R3VT77IS4WJREUOCUECGEG2CFML3QFH6GECZILMIGD2VG5TPNI4A, seg_pos={0 0}, upload=536701]: Done uploading piece. err: <nil>
Dec 31 03:55:00.858839255 (612h26m0.7299223s) [noise=true , address=145.79.227.50:4374 , piece_id=7NK5CN3C5TWMS4PAKPGVQMJTWUIBSOPC62ED4CBRHUUXABCYYSFQ, seg_pos={0 0}, upload=536701]: Done uploading piece. err: <nil>
Dec 31 03:55:00.907348172 (612h26m0.778431207s) [noise=true , address=145.79.227.38:4977 , piece_id=OJCBLWCMHRLDZNZEHIVEN4PMX424XOFN55DWOLVBQ3EDIEWYOJSA, seg_pos={0 0}, upload=536701]: Done uploading piece. err: <nil>
Dec 31 03:55:00.919182556 (612h26m0.790265591s) [noise=true , address=145.79.227.12:4767 , piece_id=5B4HV5JDM7BANTF7V6JHOG45M6ZEG5JQHW3EACHKW63QQDX5FGCA, seg_pos={0 0}, upload=536701]: Done uploading piece. err: <nil>
Dec 31 03:55:00.924445287 (612h26m0.795528332s) [noise=true , address=145.79.227.22:4511 , piece_id=BGTSHILL2Z5FSLRHZW5IOU45NW33BP7MH6LL5LAOBXQI62SUSH3Q, seg_pos={0 0}, upload=536701]: Done uploading piece. err: <nil>
Dec 31 03:55:00.926217121 (612h26m0.797300156s) [noise=true , address=145.79.227.13:4802 , piece_id=YT52KDM7B72ZP4RDGUUNK3XFTR7U4PQX27ZPQP3OUG2WDRQHSVOA, seg_pos={0 0}, upload=536701]: Done uploading piece. err: <nil>
Dec 31 03:55:00.927215392 (612h26m0.798298447s) [noise=true , address=145.79.227.26:4653 , piece_id=QXT7VXK2OMJTVEYESIJWF47S3UWOP6G5PHKDGR5HB2Q6T2CVXMPQ, seg_pos={0 0}, upload=536701]: Done uploading piece. err: <nil>
Dec 31 03:55:01.213957464 (612h26m1.085040499s) [noise=true , address=45.91.116.6:4976 , piece_id=SUSG3GFUSNXCJPYMPEXECHN4D7V7KJXB4CVNIGTF5CF5E2NT72OA, seg_pos={0 0}, upload=536701]: Done uploading piece. err: <nil>
Dec 31 03:55:01.301554410 (612h26m1.172637456s) [noise=true , address=45.91.116.11:4119 , piece_id=IXDW2KUODOF2LYPJELC2RIUS7D6RLKNTX6QOHRGZ5HWBHX7D2Q5A, seg_pos={0 0}, upload=536701]: Done uploading piece. err: <nil>
Dec 31 03:55:01.301584186 (612h26m1.172667231s) [seg_pos={0 0}, upload=536701]: Segment reached optimal threshold of 13 pieces.
Dec 31 03:55:01.301884709 (612h26m1.172967764s) [noise=true , address=77.90.31.15:4700 , piece_id=43G6PMKRQBILI5ZGYW5QJN2SEMVZDXORRGKJOT5MCDZVO22DRE6Q, seg_pos={0 0}, upload=536701]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q): protocol: expected piece hash; context canceled; EOF
Dec 31 03:55:01.301901912 (612h26m1.172984967s) [noise=true , address=77.90.31.15:4700 , piece_id=43G6PMKRQBILI5ZGYW5QJN2SEMVZDXORRGKJOT5MCDZVO22DRE6Q, seg_pos={0 0}, upload=536701]: Long tail cancellation due to optimal threshold
Dec 31 03:55:01.301916169 (612h26m1.172999214s) [noise=true , address=145.79.227.30:4762 , piece_id=D73FMYOOPAVYNACGUPAUEF6ID7ZZBQLO4QP2URVDBNDVRHCKL62A, seg_pos={0 0}, upload=536701]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: expected piece hash; context canceled; EOF
Dec 31 03:55:01.301923693 (612h26m1.173006738s) [noise=true , address=145.79.227.30:4762 , piece_id=D73FMYOOPAVYNACGUPAUEF6ID7ZZBQLO4QP2URVDBNDVRHCKL62A, seg_pos={0 0}, upload=536701]: Long tail cancellation due to optimal threshold
Dec 31 03:55:01.302109477 (612h26m1.173192523s) [noise=true , address=145.79.227.34:4905 , piece_id=O4HW7RRUM4TTXT5XXLW3MLWGETACA2VRMF5E7N2ESPHJOHEPPZ4A, seg_pos={0 0}, upload=536701]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9): protocol: expected piece hash; context canceled; EOF
Dec 31 03:55:01.302121069 (612h26m1.173204114s) [noise=true , address=145.79.227.34:4905 , piece_id=O4HW7RRUM4TTXT5XXLW3MLWGETACA2VRMF5E7N2ESPHJOHEPPZ4A, seg_pos={0 0}, upload=536701]: Long tail cancellation due to optimal threshold
Dec 31 03:55:01.302162167 (612h26m1.173245212s) [seg_pos={0 0}, upload=536701]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 31 03:55:01.302261537 (612h26m1.173344582s) [upload=536701]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 31 03:55:01.302268279 (612h26m1.173351324s) [upload=536701]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 31 03:55:01.598926553 (612h26m1.470009608s) [upload=536701]: Done uploading segments...
Dec 31 03:55:01.598964094 (612h26m1.470047149s) [upload=536701]: Upload finished. err: <nil>
Duration: 30.00310776s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [.................X....X.X..X................................] duration:
6.206055043s trace:
https://jaeger.dev.storj.io/trace/5e7c34cde28aa313
FAIL (2025-12-31 03:55:20 UTC)
orbiter warning: context deadline exceeded
Duration: 6.206283207s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
15 out of 60 is failed (threshold=6) [X.X...X...X...X.X.X...X......X..XX.....X....X............XX.] duration:
3.462143297s trace:
https://jaeger.dev.storj.io/trace/4f0364e21534b200
FAIL (2025-12-31 03:55:15 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.46217074s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [...................X........................................] duration:
15.355993906s trace:
https://jaeger.dev.storj.io/trace/634c422994952cfc
FAIL (2025-12-31 03:54:49 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 121Xa3rvgKQhHc5NBufs57Ybnw3fYKUjgRyK32TTwWoi4E8m57o, Piece ID: 7ZJODSC7K4XWDIBJGMYHMGRFCXZGHTSGWTWPO7NKPWGGT73NFGOQ) context deadline exceeded; piecestore: (Node ID: 12caEiJobigXcGWwaX3Vt16zd2NXqk1WzLWxZME44E8Q8wdhn3q, Piece ID: RTZZG5ZDBPM74ZXITWZZ6LKJCKUOQ3YY4UUTMZ5VRVKFZE4YKAUA) context deadline exceeded; piecestore: (Node ID: 1zwuzCLWy2q166Ld52WUG9fvXNdJWsQrg2NearHYg7pyxpXFen, Piece ID: EJWBT3NCXTW5JSOIICAASNNDWFMIM2AO2SAE26S5POMMLEOYUH7Q) context deadline exceeded
======LOGS======
) [noise=true , address=193.149.164.143:4704 , piece_id=XIHZWOG63PEC3AQOHHWBLF53Y3B42NOO6TRNEUR3CIJB3YZTX3LQ, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.363237580 (611h51m51.234320615s) [noise=true , address=145.79.227.15:4889 , piece_id=EACS3IK7KTZI73UJEUOZB33TZN64WDWRC7TV5YJZ5ZMQVATQ3ORQ, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.364121825 (611h51m51.23520486s) [noise=true , address=193.149.164.143:4699 , piece_id=5LCUJBU4MEYEA7CQCA2FVVX6VCEX7SU3W4N3BBPQMLP6AHG5ZKKQ, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.364545251 (611h51m51.235628286s) [noise=true , address=45.91.116.11:4143 , piece_id=RTZZG5ZDBPM74ZXITWZZ6LKJCKUOQ3YY4UUTMZ5VRVKFZE4YKAUA, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.364959129 (611h51m51.236042174s) [noise=true , address=145.79.227.3:4176 , piece_id=TBPK43ULJ75RKDV5P63GXNVYZWG7TIQ3YLKMBLU55OQ2PFGSBH7A, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.365381323 (611h51m51.236464368s) [noise=true , address=145.79.227.44:4217 , piece_id=B5JSD2DUWSMSEME5CZCWFBXELK4WRMAXLLA2QQL73SB6WR6ECXUA, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.367027327 (611h51m51.238110362s) [noise=true , address=77.90.31.11:4561 , piece_id=IRE2JNK63DRH52RXBC5D3MRL6QO5S5N44UFVFWDRNY7LZBQ3IFMQ, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.367801231 (611h51m51.238884326s) [noise=true , address=145.79.227.40:4058 , piece_id=ECUZDPLHYQW3KYSTHKWBPD3CCTUCMAZ6XMOSH5S46ACF2T5366BQ, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.369086348 (611h51m51.240169404s) [noise=true , address=77.90.31.10:4536 , piece_id=EJWBT3NCXTW5JSOIICAASNNDWFMIM2AO2SAE26S5POMMLEOYUH7Q, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.370451017 (611h51m51.241534062s) [noise=true , address=77.90.31.19:4861 , piece_id=U7I2SDPMKDL4RLP2CIWZUSCEWPM5LP67VMUOLFKBHHVSCXL55MYQ, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.371980618 (611h51m51.243063654s) [noise=true , address=64.49.13.26:4955 , piece_id=53IGVDHKGGBIEKVJ6A52Q647E74KVI4FFRQU7VYK7POFU4CCX4HA, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.374009812 (611h51m51.245092847s) [noise=true , address=77.90.31.14:4671 , piece_id=2YXJ5RJERCNTD3TJHJHF4WBHR3DEOHDIOWI53H4PDS4EWQECEU2A, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.374500948 (611h51m51.245583993s) [noise=true , address=77.90.31.13:4632 , piece_id=72ADTLC2JC7Z3Z4GYKPECNAK6QMCMONGLUKZKQFLWN5UIQ4IPZYQ, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.392662589 (611h51m51.263745634s) [noise=true , address=145.79.227.32:4865 , piece_id=ZXAGYGEC35VVET2IAX2KJII4PUK3DFQXOFAYCC5R7KV5PNQCSUIA, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.393651513 (611h51m51.264734558s) [noise=true , address=145.79.227.31:4804 , piece_id=4E3PGEBPB2BFUAHNXJI4MZFWNNNZMMFDF5UBTT2UM4F2NHOHCIQQ, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.396266312 (611h51m51.267349357s) [noise=true , address=145.79.227.29:4966 , piece_id=N4JGQH23KY2BMVZZWWX7CKZN2FC2FJBFNAZEGZ7CRMRQAYEAZJLA, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:52.828565815 (611h51m52.69964886s) [upload=536207]: Next returned nil segment
Dec 31 03:20:52.828580594 (611h51m52.699663629s) [upload=536207]: Waiting for error group managing segments...
Dec 31 03:20:53.204430167 (611h51m53.075513202s) [noise=true , address=145.79.227.15:4889 , piece_id=EACS3IK7KTZI73UJEUOZB33TZN64WDWRC7TV5YJZ5ZMQVATQ3ORQ, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:53.265960889 (611h51m53.137043924s) [noise=true , address=145.79.227.31:4804 , piece_id=4E3PGEBPB2BFUAHNXJI4MZFWNNNZMMFDF5UBTT2UM4F2NHOHCIQQ, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:53.266193351 (611h51m53.137276386s) [noise=true , address=145.79.227.32:4865 , piece_id=ZXAGYGEC35VVET2IAX2KJII4PUK3DFQXOFAYCC5R7KV5PNQCSUIA, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:53.331763405 (611h51m53.20284644s) [noise=true , address=5.231.73.2:4035 , piece_id=7ZJODSC7K4XWDIBJGMYHMGRFCXZGHTSGWTWPO7NKPWGGT73NFGOQ, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:53.412602422 (611h51m53.283685467s) [noise=true , address=77.90.31.10:4536 , piece_id=EJWBT3NCXTW5JSOIICAASNNDWFMIM2AO2SAE26S5POMMLEOYUH7Q, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:53.419806240 (611h51m53.290889275s) [noise=true , address=77.90.31.14:4671 , piece_id=2YXJ5RJERCNTD3TJHJHF4WBHR3DEOHDIOWI53H4PDS4EWQECEU2A, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:53.498590626 (611h51m53.369673671s) [noise=true , address=77.90.31.11:4561 , piece_id=IRE2JNK63DRH52RXBC5D3MRL6QO5S5N44UFVFWDRNY7LZBQ3IFMQ, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:53.520983294 (611h51m53.392066339s) [noise=true , address=45.91.116.11:4143 , piece_id=RTZZG5ZDBPM74ZXITWZZ6LKJCKUOQ3YY4UUTMZ5VRVKFZE4YKAUA, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:53.585203506 (611h51m53.456286541s) [noise=true , address=77.90.31.19:4861 , piece_id=U7I2SDPMKDL4RLP2CIWZUSCEWPM5LP67VMUOLFKBHHVSCXL55MYQ, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:54.188950073 (611h51m54.060033119s) [noise=true , address=145.79.227.44:4217 , piece_id=B5JSD2DUWSMSEME5CZCWFBXELK4WRMAXLLA2QQL73SB6WR6ECXUA, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:54.204771008 (611h51m54.075854053s) [noise=true , address=145.79.227.40:4058 , piece_id=ECUZDPLHYQW3KYSTHKWBPD3CCTUCMAZ6XMOSH5S46ACF2T5366BQ, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:54.231988834 (611h51m54.103071879s) [noise=true , address=193.149.164.143:4699 , piece_id=5LCUJBU4MEYEA7CQCA2FVVX6VCEX7SU3W4N3BBPQMLP6AHG5ZKKQ, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:54.275280076 (611h51m54.146363121s) [noise=true , address=145.79.227.29:4966 , piece_id=N4JGQH23KY2BMVZZWWX7CKZN2FC2FJBFNAZEGZ7CRMRQAYEAZJLA, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:54.307143628 (611h51m54.178226673s) [noise=true , address=77.90.31.12:4602 , piece_id=ZPVIMHS5CQAKREW456SPNTHR3WM6IWZDB6AJZIYFREALJ6V63O6A, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:54.307168195 (611h51m54.17825124s) [seg_pos={0 0}, upload=536207]: Segment reached optimal threshold of 14 pieces.
Dec 31 03:20:54.307445734 (611h51m54.178528789s) [noise=true , address=77.90.31.13:4632 , piece_id=72ADTLC2JC7Z3Z4GYKPECNAK6QMCMONGLUKZKQFLWN5UIQ4IPZYQ, seg_pos={0 0}, upload=536207]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13iPed9zYrthvh36xo3k4gsqm4tz86iVkNvAr3wauuArQLquyS): protocol: expected piece hash; context canceled; EOF
Dec 31 03:20:54.307457145 (611h51m54.1785402s) [noise=true , address=77.90.31.13:4632 , piece_id=72ADTLC2JC7Z3Z4GYKPECNAK6QMCMONGLUKZKQFLWN5UIQ4IPZYQ, seg_pos={0 0}, upload=536207]: Long tail cancellation due to optimal threshold
Dec 31 03:20:54.307494436 (611h51m54.178577471s) [noise=true , address=193.149.164.143:4704 , piece_id=XIHZWOG63PEC3AQOHHWBLF53Y3B42NOO6TRNEUR3CIJB3YZTX3LQ, seg_pos={0 0}, upload=536207]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bUVmWsfAfQiPWhDkYRNGhims8TLQqGqvhrZMYugLLTdmLk2xY): protocol: expected piece hash; context canceled; EOF
Dec 31 03:20:54.307513032 (611h51m54.178596067s) [noise=true , address=193.149.164.143:4704 , piece_id=XIHZWOG63PEC3AQOHHWBLF53Y3B42NOO6TRNEUR3CIJB3YZTX3LQ, seg_pos={0 0}, upload=536207]: Long tail cancellation due to optimal threshold
Dec 31 03:20:54.307792985 (611h51m54.17887602s) [noise=true , address=145.79.227.3:4176 , piece_id=TBPK43ULJ75RKDV5P63GXNVYZWG7TIQ3YLKMBLU55OQ2PFGSBH7A, seg_pos={0 0}, upload=536207]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kGgrsNn5tv2GkcuVHHArCACwRB2qwbj3X3JmzVF4UL6dLRW1m): protocol: write tcp 188.166.225.170:39612->145.79.227.3:4176: use of closed network connection; write tcp 188.166.225.170:39612->145.79.227.3:4176: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39612->145.79.227.3:4176: use of closed network connection
Dec 31 03:20:54.307803214 (611h51m54.178886259s) [noise=true , address=145.79.227.3:4176 , piece_id=TBPK43ULJ75RKDV5P63GXNVYZWG7TIQ3YLKMBLU55OQ2PFGSBH7A, seg_pos={0 0}, upload=536207]: Long tail cancellation due to optimal threshold
Dec 31 03:20:54.307870673 (611h51m54.178953718s) [noise=true , address=64.49.13.26:4955 , piece_id=53IGVDHKGGBIEKVJ6A52Q647E74KVI4FFRQU7VYK7POFU4CCX4HA, seg_pos={0 0}, upload=536207]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125TV3v1CzV1Q9TPfyVHsn26NZ5XcHixKxA66t96iz62FF4SHAP): protocol: expected piece hash; context canceled; EOF
Dec 31 03:20:54.307882294 (611h51m54.178965339s) [noise=true , address=64.49.13.26:4955 , piece_id=53IGVDHKGGBIEKVJ6A52Q647E74KVI4FFRQU7VYK7POFU4CCX4HA, seg_pos={0 0}, upload=536207]: Long tail cancellation due to optimal threshold
Dec 31 03:20:54.307909116 (611h51m54.178992151s) [seg_pos={0 0}, upload=536207]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 31 03:20:54.308005189 (611h51m54.179088234s) [upload=536207]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 31 03:20:54.308013014 (611h51m54.179096049s) [upload=536207]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 31 03:20:54.593642638 (611h51m54.464725683s) [upload=536207]: Done uploading segments...
Dec 31 03:20:54.593663947 (611h51m54.464746983s) [upload=536207]: Upload finished. err: <nil>
Duration: 15.356008213s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
23 out of 60 is failed (threshold=3) [.XX...X.X.......X....X..X..X....X.XXXXX.X.....X.X...XXXX..XX] duration:
30.004495279s trace:
https://jaeger.dev.storj.io/trace/2a2bd342a0f30c13
FAIL (2025-12-31 03:54:44 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks, Piece ID: AFXCQYBOSVSDPNR5MBCUPVTNYHQAVOBRUSPVYI3X4BMCJTU2M3NA) context deadline exceeded; piecestore: (Node ID: 1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, Piece ID: MZKR2I6747JZL25S6UE4H5QFL546VYVT7YN2UH2VZ3HZFOWZHAMA) context deadline exceeded
======LOGS======
Dec 31 03:54:15.622376377 (612h25m15.493459412s) [upload=536689]: Starting upload
Dec 31 03:54:15.622493540 (612h25m15.493576605s) [upload=536689]: Done starting upload
Dec 31 03:54:15.622590163 (612h25m15.493673199s) [upload=536689]: Uploading segments...
Dec 31 03:54:15.622690825 (612h25m15.49377386s) [upload=536689]: Got next segment. Inline: false
Dec 31 03:54:15.622739208 (612h25m15.493822253s) [upload=536689]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 31 03:54:15.622743576 (612h25m15.493826611s) [upload=536689]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 31 03:54:15.867798882 (612h25m15.738881927s) [seg_pos={0 0}, upload=536689]: Begin upload segment...
Dec 31 03:54:15.867848427 (612h25m15.738931472s) [seg_pos={0 0}, upload=536689]: Done begin upload segment.
Dec 31 03:54:15.867888884 (612h25m15.738971919s) [noise=true , address=77.90.31.12:4590 , piece_id=MGTB73SVWNI6L4RAGSZBOPAMFFMHOC4TSR3KCAQRFKF7KDZT6TRA, seg_pos={0 0}, upload=536689]: Uploading piece...
Dec 31 03:54:15.867967914 (612h25m15.739050979s) [noise=true , address=145.79.227.48:4303 , piece_id=UMBQHOJGWFC55OVPWADJT62EXKFCJIMD5U4SMPDX2VATGTCYFGJQ, seg_pos={0 0}, upload=536689]: Uploading piece...
Dec 31 03:54:15.868448419 (612h25m15.739531464s) [noise=true , address=145.79.227.2:4115 , piece_id=ULGPBAPHSLGWMRMJUBCEOCOYOBVPX7YFKRA3MEGNCUE7A63HQA5A, seg_pos={0 0}, upload=536689]: Uploading piece...
Dec 31 03:54:15.868739012 (612h25m15.739822047s) [noise=true , address=145.79.227.16:4910 , piece_id=USSNA7CL5QFGY4SJGQJ72LFKXRMIO575IFTP4LJMD4HZ76RIYN2Q, seg_pos={0 0}, upload=536689]: Uploading piece...
Dec 31 03:54:15.869092176 (612h25m15.740175211s) [noise=true , address=145.79.227.22:4511 , piece_id=M3IMD33ARKRT2PC5GVTHUOAW7TM5MZ7ETXP55ZFRHASH3OXWOHLA, seg_pos={0 0}, upload=536689]: Uploading piece...
Dec 31 03:54:15.869427814 (612h25m15.740510849s) [noise=true , address=77.90.31.17:4770 , piece_id=VTX7776QNOAROJCDUEYOATMKWBRZV5A2BI2B46YADR2UYUWB2TIQ, seg_pos={0 0}, upload=536689]: Uploading piece...
Dec 31 03:54:15.869727425 (612h25m15.74081046s) [noise=true , address=145.79.227.50:4373 , piece_id=QOBSR7NUEKWAEBO3E2QFBLGQ5SPTDYGTDMDHJZZ2LFNHHGT42XNA, seg_pos={0 0}, upload=536689]: Uploading piece...
Dec 31 03:54:15.870075929 (612h25m15.741158974s) [noise=true , address=145.79.227.36:4517 , piece_id=IH2TVQ45OBPDBG2PMN3MF5NWTE3H2EPJD6SJIIKNHOUHCDOZ2UZA, seg_pos={0 0}, upload=536689]: Uploading piece...
Dec 31 03:54:15.870516206 (612h25m15.741599241s) [noise=true , address=77.90.31.10:4518 , piece_id=CZAYO2TN6XJXASLTBJ5GZQVKZMXQEUCH3776UVFI2BJCMODKTOPQ, seg_pos={0 0}, upload=536689]: Uploading piece...
Dec 31 03:54:15.871668621 (612h25m15.742751656s) [noise=true , address=145.79.227.6:4257 , piece_id=D2XLSR6HBVKADHJJSWGKMRBNUW7ZF2XHOL3DG74G5FRX4UZ3RIEQ, seg_pos={0 0}, upload=536689]: Uploading piece...
Dec 31 03:54:15.872875439 (612h25m15.743958474s) [noise=true , address=145.79.227.40:4050 , piece_id=ZI3XCT5HEZV2FJ6GRAOO3JKAYZQJMU3IWHM6PZSD6PTTVG6TNBKQ, seg_pos={0 0}, upload=536689]: Uploading piece...
Dec 31 03:54:15.874290813 (612h25m15.745373859s) [noise=true , address=45.91.116.8:4048 , piece_id=MZKR2I6747JZL25S6UE4H5QFL546VYVT7YN2UH2VZ3HZFOWZHAMA, seg_pos={0 0}, upload=536689]: Uploading piece...
Dec 31 03:54:15.874711495 (612h25m15.74579453s) [noise=true , address=77.90.31.11:4556 , piece_id=AFXCQYBOSVSDPNR5MBCUPVTNYHQAVOBRUSPVYI3X4BMCJTU2M3NA, seg_pos={0 0}, upload=536689]: Uploading piece...
Dec 31 03:54:15.875290137 (612h25m15.746373182s) [noise=true , address=77.90.31.21:4880 , piece_id=7FCTBQMDFSWC6JFP62XZ2T3DOH7BCOWZFEWVABQGAO2FBVZM3WDA, seg_pos={0 0}, upload=536689]: Uploading piece...
Dec 31 03:54:15.878274500 (612h25m15.749357545s) [noise=true , address=45.91.116.5:4300 , piece_id=AZ246F4TQGR6FFQCDV574TDONZVC2PDOIP6QGJVZO4BDVIZTSFSQ, seg_pos={0 0}, upload=536689]: Uploading piece...
Dec 31 03:54:15.881893350 (612h25m15.752976395s) [noise=true , address=145.79.227.15:4875 , piece_id=CKVJKQPO6UXCYA6KGR3TIZE3EIJW7LCO6BZLXUAGZ2ECEASDNFHQ, seg_pos={0 0}, upload=536689]: Uploading piece...
Dec 31 03:54:17.469152428 (612h25m17.340235463s) [upload=536689]: Next returned nil segment
Dec 31 03:54:17.469166955 (612h25m17.340249991s) [upload=536689]: Waiting for error group managing segments...
Dec 31 03:54:17.860849298 (612h25m17.731932343s) [noise=true , address=145.79.227.22:4511 , piece_id=M3IMD33ARKRT2PC5GVTHUOAW7TM5MZ7ETXP55ZFRHASH3OXWOHLA, seg_pos={0 0}, upload=536689]: Done uploading piece. err: <nil>
Dec 31 03:54:17.862641900 (612h25m17.733724946s) [noise=true , address=145.79.227.36:4517 , piece_id=IH2TVQ45OBPDBG2PMN3MF5NWTE3H2EPJD6SJIIKNHOUHCDOZ2UZA, seg_pos={0 0}, upload=536689]: Done uploading piece. err: <nil>
Dec 31 03:54:17.866808555 (612h25m17.73789159s) [noise=true , address=145.79.227.50:4373 , piece_id=QOBSR7NUEKWAEBO3E2QFBLGQ5SPTDYGTDMDHJZZ2LFNHHGT42XNA, seg_pos={0 0}, upload=536689]: Done uploading piece. err: <nil>
Dec 31 03:54:17.875463294 (612h25m17.746546329s) [noise=true , address=145.79.227.2:4115 , piece_id=ULGPBAPHSLGWMRMJUBCEOCOYOBVPX7YFKRA3MEGNCUE7A63HQA5A, seg_pos={0 0}, upload=536689]: Done uploading piece. err: <nil>
Dec 31 03:54:17.893494217 (612h25m17.764577262s) [noise=true , address=145.79.227.15:4875 , piece_id=CKVJKQPO6UXCYA6KGR3TIZE3EIJW7LCO6BZLXUAGZ2ECEASDNFHQ, seg_pos={0 0}, upload=536689]: Done uploading piece. err: <nil>
Dec 31 03:54:17.941639081 (612h25m17.812722126s) [noise=true , address=145.79.227.16:4910 , piece_id=USSNA7CL5QFGY4SJGQJ72LFKXRMIO575IFTP4LJMD4HZ76RIYN2Q, seg_pos={0 0}, upload=536689]: Done uploading piece. err: <nil>
Dec 31 03:54:17.942557119 (612h25m17.813640164s) [noise=true , address=145.79.227.48:4303 , piece_id=UMBQHOJGWFC55OVPWADJT62EXKFCJIMD5U4SMPDX2VATGTCYFGJQ, seg_pos={0 0}, upload=536689]: Done uploading piece. err: <nil>
Dec 31 03:54:17.953017327 (612h25m17.824100362s) [noise=true , address=145.79.227.6:4257 , piece_id=D2XLSR6HBVKADHJJSWGKMRBNUW7ZF2XHOL3DG74G5FRX4UZ3RIEQ, seg_pos={0 0}, upload=536689]: Done uploading piece. err: <nil>
Dec 31 03:54:18.154920886 (612h25m18.026003921s) [noise=true , address=77.90.31.12:4590 , piece_id=MGTB73SVWNI6L4RAGSZBOPAMFFMHOC4TSR3KCAQRFKF7KDZT6TRA, seg_pos={0 0}, upload=536689]: Done uploading piece. err: <nil>
Dec 31 03:54:18.172881796 (612h25m18.043964831s) [noise=true , address=77.90.31.10:4518 , piece_id=CZAYO2TN6XJXASLTBJ5GZQVKZMXQEUCH3776UVFI2BJCMODKTOPQ, seg_pos={0 0}, upload=536689]: Done uploading piece. err: <nil>
Dec 31 03:54:18.176619102 (612h25m18.047702147s) [noise=true , address=77.90.31.11:4556 , piece_id=AFXCQYBOSVSDPNR5MBCUPVTNYHQAVOBRUSPVYI3X4BMCJTU2M3NA, seg_pos={0 0}, upload=536689]: Done uploading piece. err: <nil>
Dec 31 03:54:18.242978308 (612h25m18.114061343s) [noise=true , address=45.91.116.8:4048 , piece_id=MZKR2I6747JZL25S6UE4H5QFL546VYVT7YN2UH2VZ3HZFOWZHAMA, seg_pos={0 0}, upload=536689]: Done uploading piece. err: <nil>
Dec 31 03:54:18.530722799 (612h25m18.401805844s) [noise=true , address=45.91.116.5:4300 , piece_id=AZ246F4TQGR6FFQCDV574TDONZVC2PDOIP6QGJVZO4BDVIZTSFSQ, seg_pos={0 0}, upload=536689]: Done uploading piece. err: <nil>
Dec 31 03:54:18.530747587 (612h25m18.401830642s) [seg_pos={0 0}, upload=536689]: Segment reached optimal threshold of 13 pieces.
Dec 31 03:54:18.530988956 (612h25m18.402072001s) [noise=true , address=77.90.31.21:4880 , piece_id=7FCTBQMDFSWC6JFP62XZ2T3DOH7BCOWZFEWVABQGAO2FBVZM3WDA, seg_pos={0 0}, upload=536689]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb): protocol: expected piece hash; context canceled; EOF
Dec 31 03:54:18.531002301 (612h25m18.402085346s) [noise=true , address=77.90.31.21:4880 , piece_id=7FCTBQMDFSWC6JFP62XZ2T3DOH7BCOWZFEWVABQGAO2FBVZM3WDA, seg_pos={0 0}, upload=536689]: Long tail cancellation due to optimal threshold
Dec 31 03:54:18.531074529 (612h25m18.402157574s) [noise=true , address=145.79.227.40:4050 , piece_id=ZI3XCT5HEZV2FJ6GRAOO3JKAYZQJMU3IWHM6PZSD6PTTVG6TNBKQ, seg_pos={0 0}, upload=536689]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64): protocol: expected piece hash; context canceled; EOF
Dec 31 03:54:18.531086060 (612h25m18.402169105s) [noise=true , address=145.79.227.40:4050 , piece_id=ZI3XCT5HEZV2FJ6GRAOO3JKAYZQJMU3IWHM6PZSD6PTTVG6TNBKQ, seg_pos={0 0}, upload=536689]: Long tail cancellation due to optimal threshold
Dec 31 03:54:18.531294888 (612h25m18.402377933s) [noise=true , address=77.90.31.17:4770 , piece_id=VTX7776QNOAROJCDUEYOATMKWBRZV5A2BI2B46YADR2UYUWB2TIQ, seg_pos={0 0}, upload=536689]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: expected piece hash; context canceled; EOF
Dec 31 03:54:18.531303686 (612h25m18.402386731s) [noise=true , address=77.90.31.17:4770 , piece_id=VTX7776QNOAROJCDUEYOATMKWBRZV5A2BI2B46YADR2UYUWB2TIQ, seg_pos={0 0}, upload=536689]: Long tail cancellation due to optimal threshold
Dec 31 03:54:18.531323382 (612h25m18.402406428s) [seg_pos={0 0}, upload=536689]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 31 03:54:18.531407041 (612h25m18.402490096s) [upload=536689]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 31 03:54:18.531425828 (612h25m18.402508883s) [upload=536689]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 31 03:54:18.831057925 (612h25m18.70214097s) [upload=536689]: Done uploading segments...
Dec 31 03:54:18.831075368 (612h25m18.702158403s) [upload=536689]: Upload finished. err: <nil>
Duration: 30.004518404s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [....................X.......................................] duration:
33.925092898s trace:
https://jaeger.dev.storj.io/trace/634c422994952cfc
FAIL (2025-12-31 03:54:33 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 121Xa3rvgKQhHc5NBufs57Ybnw3fYKUjgRyK32TTwWoi4E8m57o, Piece ID: 7ZJODSC7K4XWDIBJGMYHMGRFCXZGHTSGWTWPO7NKPWGGT73NFGOQ) context deadline exceeded; piecestore: (Node ID: 12caEiJobigXcGWwaX3Vt16zd2NXqk1WzLWxZME44E8Q8wdhn3q, Piece ID: RTZZG5ZDBPM74ZXITWZZ6LKJCKUOQ3YY4UUTMZ5VRVKFZE4YKAUA) context deadline exceeded; piecestore: (Node ID: 1zwuzCLWy2q166Ld52WUG9fvXNdJWsQrg2NearHYg7pyxpXFen, Piece ID: EJWBT3NCXTW5JSOIICAASNNDWFMIM2AO2SAE26S5POMMLEOYUH7Q) context deadline exceeded
======LOGS======
) [noise=true , address=193.149.164.143:4704 , piece_id=XIHZWOG63PEC3AQOHHWBLF53Y3B42NOO6TRNEUR3CIJB3YZTX3LQ, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.363237580 (611h51m51.234320615s) [noise=true , address=145.79.227.15:4889 , piece_id=EACS3IK7KTZI73UJEUOZB33TZN64WDWRC7TV5YJZ5ZMQVATQ3ORQ, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.364121825 (611h51m51.23520486s) [noise=true , address=193.149.164.143:4699 , piece_id=5LCUJBU4MEYEA7CQCA2FVVX6VCEX7SU3W4N3BBPQMLP6AHG5ZKKQ, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.364545251 (611h51m51.235628286s) [noise=true , address=45.91.116.11:4143 , piece_id=RTZZG5ZDBPM74ZXITWZZ6LKJCKUOQ3YY4UUTMZ5VRVKFZE4YKAUA, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.364959129 (611h51m51.236042174s) [noise=true , address=145.79.227.3:4176 , piece_id=TBPK43ULJ75RKDV5P63GXNVYZWG7TIQ3YLKMBLU55OQ2PFGSBH7A, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.365381323 (611h51m51.236464368s) [noise=true , address=145.79.227.44:4217 , piece_id=B5JSD2DUWSMSEME5CZCWFBXELK4WRMAXLLA2QQL73SB6WR6ECXUA, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.367027327 (611h51m51.238110362s) [noise=true , address=77.90.31.11:4561 , piece_id=IRE2JNK63DRH52RXBC5D3MRL6QO5S5N44UFVFWDRNY7LZBQ3IFMQ, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.367801231 (611h51m51.238884326s) [noise=true , address=145.79.227.40:4058 , piece_id=ECUZDPLHYQW3KYSTHKWBPD3CCTUCMAZ6XMOSH5S46ACF2T5366BQ, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.369086348 (611h51m51.240169404s) [noise=true , address=77.90.31.10:4536 , piece_id=EJWBT3NCXTW5JSOIICAASNNDWFMIM2AO2SAE26S5POMMLEOYUH7Q, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.370451017 (611h51m51.241534062s) [noise=true , address=77.90.31.19:4861 , piece_id=U7I2SDPMKDL4RLP2CIWZUSCEWPM5LP67VMUOLFKBHHVSCXL55MYQ, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.371980618 (611h51m51.243063654s) [noise=true , address=64.49.13.26:4955 , piece_id=53IGVDHKGGBIEKVJ6A52Q647E74KVI4FFRQU7VYK7POFU4CCX4HA, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.374009812 (611h51m51.245092847s) [noise=true , address=77.90.31.14:4671 , piece_id=2YXJ5RJERCNTD3TJHJHF4WBHR3DEOHDIOWI53H4PDS4EWQECEU2A, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.374500948 (611h51m51.245583993s) [noise=true , address=77.90.31.13:4632 , piece_id=72ADTLC2JC7Z3Z4GYKPECNAK6QMCMONGLUKZKQFLWN5UIQ4IPZYQ, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.392662589 (611h51m51.263745634s) [noise=true , address=145.79.227.32:4865 , piece_id=ZXAGYGEC35VVET2IAX2KJII4PUK3DFQXOFAYCC5R7KV5PNQCSUIA, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.393651513 (611h51m51.264734558s) [noise=true , address=145.79.227.31:4804 , piece_id=4E3PGEBPB2BFUAHNXJI4MZFWNNNZMMFDF5UBTT2UM4F2NHOHCIQQ, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.396266312 (611h51m51.267349357s) [noise=true , address=145.79.227.29:4966 , piece_id=N4JGQH23KY2BMVZZWWX7CKZN2FC2FJBFNAZEGZ7CRMRQAYEAZJLA, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:52.828565815 (611h51m52.69964886s) [upload=536207]: Next returned nil segment
Dec 31 03:20:52.828580594 (611h51m52.699663629s) [upload=536207]: Waiting for error group managing segments...
Dec 31 03:20:53.204430167 (611h51m53.075513202s) [noise=true , address=145.79.227.15:4889 , piece_id=EACS3IK7KTZI73UJEUOZB33TZN64WDWRC7TV5YJZ5ZMQVATQ3ORQ, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:53.265960889 (611h51m53.137043924s) [noise=true , address=145.79.227.31:4804 , piece_id=4E3PGEBPB2BFUAHNXJI4MZFWNNNZMMFDF5UBTT2UM4F2NHOHCIQQ, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:53.266193351 (611h51m53.137276386s) [noise=true , address=145.79.227.32:4865 , piece_id=ZXAGYGEC35VVET2IAX2KJII4PUK3DFQXOFAYCC5R7KV5PNQCSUIA, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:53.331763405 (611h51m53.20284644s) [noise=true , address=5.231.73.2:4035 , piece_id=7ZJODSC7K4XWDIBJGMYHMGRFCXZGHTSGWTWPO7NKPWGGT73NFGOQ, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:53.412602422 (611h51m53.283685467s) [noise=true , address=77.90.31.10:4536 , piece_id=EJWBT3NCXTW5JSOIICAASNNDWFMIM2AO2SAE26S5POMMLEOYUH7Q, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:53.419806240 (611h51m53.290889275s) [noise=true , address=77.90.31.14:4671 , piece_id=2YXJ5RJERCNTD3TJHJHF4WBHR3DEOHDIOWI53H4PDS4EWQECEU2A, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:53.498590626 (611h51m53.369673671s) [noise=true , address=77.90.31.11:4561 , piece_id=IRE2JNK63DRH52RXBC5D3MRL6QO5S5N44UFVFWDRNY7LZBQ3IFMQ, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:53.520983294 (611h51m53.392066339s) [noise=true , address=45.91.116.11:4143 , piece_id=RTZZG5ZDBPM74ZXITWZZ6LKJCKUOQ3YY4UUTMZ5VRVKFZE4YKAUA, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:53.585203506 (611h51m53.456286541s) [noise=true , address=77.90.31.19:4861 , piece_id=U7I2SDPMKDL4RLP2CIWZUSCEWPM5LP67VMUOLFKBHHVSCXL55MYQ, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:54.188950073 (611h51m54.060033119s) [noise=true , address=145.79.227.44:4217 , piece_id=B5JSD2DUWSMSEME5CZCWFBXELK4WRMAXLLA2QQL73SB6WR6ECXUA, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:54.204771008 (611h51m54.075854053s) [noise=true , address=145.79.227.40:4058 , piece_id=ECUZDPLHYQW3KYSTHKWBPD3CCTUCMAZ6XMOSH5S46ACF2T5366BQ, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:54.231988834 (611h51m54.103071879s) [noise=true , address=193.149.164.143:4699 , piece_id=5LCUJBU4MEYEA7CQCA2FVVX6VCEX7SU3W4N3BBPQMLP6AHG5ZKKQ, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:54.275280076 (611h51m54.146363121s) [noise=true , address=145.79.227.29:4966 , piece_id=N4JGQH23KY2BMVZZWWX7CKZN2FC2FJBFNAZEGZ7CRMRQAYEAZJLA, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:54.307143628 (611h51m54.178226673s) [noise=true , address=77.90.31.12:4602 , piece_id=ZPVIMHS5CQAKREW456SPNTHR3WM6IWZDB6AJZIYFREALJ6V63O6A, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:54.307168195 (611h51m54.17825124s) [seg_pos={0 0}, upload=536207]: Segment reached optimal threshold of 14 pieces.
Dec 31 03:20:54.307445734 (611h51m54.178528789s) [noise=true , address=77.90.31.13:4632 , piece_id=72ADTLC2JC7Z3Z4GYKPECNAK6QMCMONGLUKZKQFLWN5UIQ4IPZYQ, seg_pos={0 0}, upload=536207]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13iPed9zYrthvh36xo3k4gsqm4tz86iVkNvAr3wauuArQLquyS): protocol: expected piece hash; context canceled; EOF
Dec 31 03:20:54.307457145 (611h51m54.1785402s) [noise=true , address=77.90.31.13:4632 , piece_id=72ADTLC2JC7Z3Z4GYKPECNAK6QMCMONGLUKZKQFLWN5UIQ4IPZYQ, seg_pos={0 0}, upload=536207]: Long tail cancellation due to optimal threshold
Dec 31 03:20:54.307494436 (611h51m54.178577471s) [noise=true , address=193.149.164.143:4704 , piece_id=XIHZWOG63PEC3AQOHHWBLF53Y3B42NOO6TRNEUR3CIJB3YZTX3LQ, seg_pos={0 0}, upload=536207]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bUVmWsfAfQiPWhDkYRNGhims8TLQqGqvhrZMYugLLTdmLk2xY): protocol: expected piece hash; context canceled; EOF
Dec 31 03:20:54.307513032 (611h51m54.178596067s) [noise=true , address=193.149.164.143:4704 , piece_id=XIHZWOG63PEC3AQOHHWBLF53Y3B42NOO6TRNEUR3CIJB3YZTX3LQ, seg_pos={0 0}, upload=536207]: Long tail cancellation due to optimal threshold
Dec 31 03:20:54.307792985 (611h51m54.17887602s) [noise=true , address=145.79.227.3:4176 , piece_id=TBPK43ULJ75RKDV5P63GXNVYZWG7TIQ3YLKMBLU55OQ2PFGSBH7A, seg_pos={0 0}, upload=536207]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kGgrsNn5tv2GkcuVHHArCACwRB2qwbj3X3JmzVF4UL6dLRW1m): protocol: write tcp 188.166.225.170:39612->145.79.227.3:4176: use of closed network connection; write tcp 188.166.225.170:39612->145.79.227.3:4176: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39612->145.79.227.3:4176: use of closed network connection
Dec 31 03:20:54.307803214 (611h51m54.178886259s) [noise=true , address=145.79.227.3:4176 , piece_id=TBPK43ULJ75RKDV5P63GXNVYZWG7TIQ3YLKMBLU55OQ2PFGSBH7A, seg_pos={0 0}, upload=536207]: Long tail cancellation due to optimal threshold
Dec 31 03:20:54.307870673 (611h51m54.178953718s) [noise=true , address=64.49.13.26:4955 , piece_id=53IGVDHKGGBIEKVJ6A52Q647E74KVI4FFRQU7VYK7POFU4CCX4HA, seg_pos={0 0}, upload=536207]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125TV3v1CzV1Q9TPfyVHsn26NZ5XcHixKxA66t96iz62FF4SHAP): protocol: expected piece hash; context canceled; EOF
Dec 31 03:20:54.307882294 (611h51m54.178965339s) [noise=true , address=64.49.13.26:4955 , piece_id=53IGVDHKGGBIEKVJ6A52Q647E74KVI4FFRQU7VYK7POFU4CCX4HA, seg_pos={0 0}, upload=536207]: Long tail cancellation due to optimal threshold
Dec 31 03:20:54.307909116 (611h51m54.178992151s) [seg_pos={0 0}, upload=536207]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 31 03:20:54.308005189 (611h51m54.179088234s) [upload=536207]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 31 03:20:54.308013014 (611h51m54.179096049s) [upload=536207]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 31 03:20:54.593642638 (611h51m54.464725683s) [upload=536207]: Done uploading segments...
Dec 31 03:20:54.593663947 (611h51m54.464746983s) [upload=536207]: Upload finished. err: <nil>
Duration: 33.92511015s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [..................X....X.X..X...............................] duration:
18.131650507s trace:
https://jaeger.dev.storj.io/trace/5e7c34cde28aa313
FAIL (2025-12-31 03:54:33 UTC)
orbiter warning: context deadline exceeded
Duration: 18.131680475s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
15 out of 60 is failed (threshold=6) [.X.X...X...X...X.X.X...X......X..XX.....X....X............XX] duration:
5.001608224s trace:
https://jaeger.dev.storj.io/trace/4f0364e21534b200
FAIL (2025-12-31 03:54:33 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001630707s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
23 out of 60 is failed (threshold=3) [X.XX...X.X.......X....X..X..X....X.XXXXX.X.....X.X...XXXX..X] duration:
30.003075703s trace:
https://jaeger.dev.storj.io/trace/6d6c3dd3612728b2
FAIL (2025-12-31 03:54:14 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
03:53:44.919428273 (612h24m44.790511328s) [upload=536680]: Starting upload
Dec 31 03:53:44.919533262 (612h24m44.790616297s) [upload=536680]: Done starting upload
Dec 31 03:53:44.919610219 (612h24m44.790693254s) [upload=536680]: Uploading segments...
Dec 31 03:53:44.919836940 (612h24m44.790919975s) [upload=536680]: Got next segment. Inline: false
Dec 31 03:53:44.919916242 (612h24m44.790999277s) [upload=536680]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 31 03:53:44.919922934 (612h24m44.791005979s) [upload=536680]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 31 03:53:45.167748454 (612h24m45.038831499s) [seg_pos={0 0}, upload=536680]: Begin upload segment...
Dec 31 03:53:45.167802838 (612h24m45.038885873s) [seg_pos={0 0}, upload=536680]: Done begin upload segment.
Dec 31 03:53:45.167845489 (612h24m45.038928534s) [noise=true , address=145.79.227.38:4978 , piece_id=6PRKL2KTTYRPPNPHOL3324JZUI3YPCDJ47XLBPYVSBTOSDD6H6VA, seg_pos={0 0}, upload=536680]: Uploading piece...
Dec 31 03:53:45.170058192 (612h24m45.041141227s) [noise=true , address=45.91.116.6:4976 , piece_id=QBO6PESXU3YQNMTGK3WRI572VKQ3XE6XVJTHDKOXNI2KXU6K36PQ, seg_pos={0 0}, upload=536680]: Uploading piece...
Dec 31 03:53:45.170193239 (612h24m45.041276274s) [noise=true , address=145.79.227.27:4689 , piece_id=XILNDZ6PAIHVPV6WN5REX2UBNCWMR5COPSOSTWWUWHJMRBOH7T5A, seg_pos={0 0}, upload=536680]: Uploading piece...
Dec 31 03:53:45.171186010 (612h24m45.042269045s) [noise=true , address=145.79.227.46:4603 , piece_id=OOWHRAG57KRSXPS2ZUUPO2EDGWZ32UJV2TS6IUBJ4ZVLFEECT2QQ, seg_pos={0 0}, upload=536680]: Uploading piece...
Dec 31 03:53:45.171439362 (612h24m45.042522397s) [noise=true , address=45.91.116.10:4083 , piece_id=LKI324QXO7PBDIPVJUXA5UKBX2OBSWF6MY6M23SAODBPUOSNZE6A, seg_pos={0 0}, upload=536680]: Uploading piece...
Dec 31 03:53:45.171882335 (612h24m45.042965371s) [noise=true , address=77.90.31.16:4736 , piece_id=BIQ7HHOZHM3KZX2CGIDKCAZN7ARH2MYDHW7OQBZJE6WZI7C4J6ZA, seg_pos={0 0}, upload=536680]: Uploading piece...
Dec 31 03:53:45.172204198 (612h24m45.043287233s) [noise=true , address=77.90.31.10:4517 , piece_id=4OOBIXZDBMKZMN72HGAL36T67VGU5M6HPFZLS6N5E4D5JITFUU4Q, seg_pos={0 0}, upload=536680]: Uploading piece...
Dec 31 03:53:45.172330729 (612h24m45.043413775s) [noise=true , address=77.90.31.18:4805 , piece_id=LHPBRVN43K7GQLT56D34WANPMLBPGUMP57VAMCKSUV4QOXKPOJEQ, seg_pos={0 0}, upload=536680]: Uploading piece...
Dec 31 03:53:45.172668472 (612h24m45.043751517s) [noise=true , address=145.79.227.26:4653 , piece_id=HE3T2RRZ6OPJK375Q5VKW5CP44LUCQA5CA5ZYHGIIPBNE5FWWTDQ, seg_pos={0 0}, upload=536680]: Uploading piece...
Dec 31 03:53:45.172847002 (612h24m45.043930037s) [noise=true , address=145.79.227.36:4517 , piece_id=YAVVJVFQH6LPBWCNYCKES742OQPNVKEY6P2PTKVWWF7SSF62NRLQ, seg_pos={0 0}, upload=536680]: Uploading piece...
Dec 31 03:53:45.173095515 (612h24m45.04417855s) [noise=true , address=45.91.116.11:4119 , piece_id=PWPLHWPLMUYADVQ5WLZPYSC4PVPR6X55ZCKBEO7MDFSNFOMFZ6ZA, seg_pos={0 0}, upload=536680]: Uploading piece...
Dec 31 03:53:45.173260439 (612h24m45.044343474s) [noise=true , address=145.79.227.10:4402 , piece_id=B7ZOGHGHCISE2EV6P6XLOVT3FMCO6AIZBYSFFF64UCXU6HMMIRPQ, seg_pos={0 0}, upload=536680]: Uploading piece...
Dec 31 03:53:45.173470559 (612h24m45.044553594s) [noise=true , address=145.79.227.45:4231 , piece_id=HOMAULZWVE3YXHDFJRDQMMFTHKXJHP5IDKUKBIOC2HUXCFQ5MHSA, seg_pos={0 0}, upload=536680]: Uploading piece...
Dec 31 03:53:45.173647626 (612h24m45.044730671s) [noise=true , address=145.79.227.37:4565 , piece_id=VAEITELJNTO6SFMPL7VIZVPVZKFRZAHNEIWZRGZF5VMHRBISYT3A, seg_pos={0 0}, upload=536680]: Uploading piece...
Dec 31 03:53:45.173845624 (612h24m45.044928659s) [noise=true , address=145.79.227.51:4410 , piece_id=ONZPH5DORAXJ353CXLC2UH5Y22LMU4UCAXS3IH4MKGYHKXMSVRBQ, seg_pos={0 0}, upload=536680]: Uploading piece...
Dec 31 03:53:45.174065602 (612h24m45.045148637s) [noise=true , address=45.91.116.7:4011 , piece_id=3OD6IT3CMMJFAFRYDLCLOXWBKRJ35WLQUI3FKJCFCSZ2ZTOLMXOQ, seg_pos={0 0}, upload=536680]: Uploading piece...
Dec 31 03:53:46.872340207 (612h24m46.743423252s) [upload=536680]: Next returned nil segment
Dec 31 03:53:46.872356809 (612h24m46.743439875s) [upload=536680]: Waiting for error group managing segments...
Dec 31 03:53:47.152385548 (612h24m47.023468583s) [noise=true , address=145.79.227.37:4565 , piece_id=VAEITELJNTO6SFMPL7VIZVPVZKFRZAHNEIWZRGZF5VMHRBISYT3A, seg_pos={0 0}, upload=536680]: Done uploading piece. err: <nil>
Dec 31 03:53:47.152457325 (612h24m47.02354036s) [noise=true , address=145.79.227.51:4410 , piece_id=ONZPH5DORAXJ353CXLC2UH5Y22LMU4UCAXS3IH4MKGYHKXMSVRBQ, seg_pos={0 0}, upload=536680]: Done uploading piece. err: <nil>
Dec 31 03:53:47.155444473 (612h24m47.026527508s) [noise=true , address=145.79.227.10:4402 , piece_id=B7ZOGHGHCISE2EV6P6XLOVT3FMCO6AIZBYSFFF64UCXU6HMMIRPQ, seg_pos={0 0}, upload=536680]: Done uploading piece. err: <nil>
Dec 31 03:53:47.156474354 (612h24m47.027557399s) [noise=true , address=145.79.227.26:4653 , piece_id=HE3T2RRZ6OPJK375Q5VKW5CP44LUCQA5CA5ZYHGIIPBNE5FWWTDQ, seg_pos={0 0}, upload=536680]: Done uploading piece. err: <nil>
Dec 31 03:53:47.160317441 (612h24m47.031400476s) [noise=true , address=145.79.227.46:4603 , piece_id=OOWHRAG57KRSXPS2ZUUPO2EDGWZ32UJV2TS6IUBJ4ZVLFEECT2QQ, seg_pos={0 0}, upload=536680]: Done uploading piece. err: <nil>
Dec 31 03:53:47.161533517 (612h24m47.032616552s) [noise=true , address=145.79.227.45:4231 , piece_id=HOMAULZWVE3YXHDFJRDQMMFTHKXJHP5IDKUKBIOC2HUXCFQ5MHSA, seg_pos={0 0}, upload=536680]: Done uploading piece. err: <nil>
Dec 31 03:53:47.180017402 (612h24m47.051100437s) [noise=true , address=145.79.227.38:4978 , piece_id=6PRKL2KTTYRPPNPHOL3324JZUI3YPCDJ47XLBPYVSBTOSDD6H6VA, seg_pos={0 0}, upload=536680]: Done uploading piece. err: <nil>
Dec 31 03:53:47.193761172 (612h24m47.064844207s) [noise=true , address=145.79.227.36:4517 , piece_id=YAVVJVFQH6LPBWCNYCKES742OQPNVKEY6P2PTKVWWF7SSF62NRLQ, seg_pos={0 0}, upload=536680]: Done uploading piece. err: <nil>
Dec 31 03:53:47.338749838 (612h24m47.209832873s) [noise=true , address=145.79.227.27:4689 , piece_id=XILNDZ6PAIHVPV6WN5REX2UBNCWMR5COPSOSTWWUWHJMRBOH7T5A, seg_pos={0 0}, upload=536680]: Done uploading piece. err: <nil>
Dec 31 03:53:47.369941552 (612h24m47.241024597s) [noise=true , address=77.90.31.18:4805 , piece_id=LHPBRVN43K7GQLT56D34WANPMLBPGUMP57VAMCKSUV4QOXKPOJEQ, seg_pos={0 0}, upload=536680]: Done uploading piece. err: <nil>
Dec 31 03:53:47.442575396 (612h24m47.313658431s) [noise=true , address=77.90.31.10:4517 , piece_id=4OOBIXZDBMKZMN72HGAL36T67VGU5M6HPFZLS6N5E4D5JITFUU4Q, seg_pos={0 0}, upload=536680]: Done uploading piece. err: <nil>
Dec 31 03:53:47.486367121 (612h24m47.357450166s) [noise=true , address=77.90.31.16:4736 , piece_id=BIQ7HHOZHM3KZX2CGIDKCAZN7ARH2MYDHW7OQBZJE6WZI7C4J6ZA, seg_pos={0 0}, upload=536680]: Done uploading piece. err: <nil>
Dec 31 03:53:47.546244033 (612h24m47.417327068s) [noise=true , address=45.91.116.7:4011 , piece_id=3OD6IT3CMMJFAFRYDLCLOXWBKRJ35WLQUI3FKJCFCSZ2ZTOLMXOQ, seg_pos={0 0}, upload=536680]: Done uploading piece. err: <nil>
Dec 31 03:53:47.546264111 (612h24m47.417347146s) [seg_pos={0 0}, upload=536680]: Segment reached optimal threshold of 13 pieces.
Dec 31 03:53:47.546495762 (612h24m47.417578807s) [noise=true , address=45.91.116.11:4119 , piece_id=PWPLHWPLMUYADVQ5WLZPYSC4PVPR6X55ZCKBEO7MDFSNFOMFZ6ZA, seg_pos={0 0}, upload=536680]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: expected piece hash; context canceled; EOF
Dec 31 03:53:47.546503878 (612h24m47.417586923s) [noise=true , address=45.91.116.11:4119 , piece_id=PWPLHWPLMUYADVQ5WLZPYSC4PVPR6X55ZCKBEO7MDFSNFOMFZ6ZA, seg_pos={0 0}, upload=536680]: Long tail cancellation due to optimal threshold
Dec 31 03:53:47.546620389 (612h24m47.417703424s) [noise=true , address=45.91.116.6:4976 , piece_id=QBO6PESXU3YQNMTGK3WRI572VKQ3XE6XVJTHDKOXNI2KXU6K36PQ, seg_pos={0 0}, upload=536680]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp): protocol: write tcp 188.166.225.170:38392->45.91.116.6:4976: use of closed network connection; write tcp 188.166.225.170:38392->45.91.116.6:4976: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38392->45.91.116.6:4976: use of closed network connection
Dec 31 03:53:47.546626361 (612h24m47.417709396s) [noise=true , address=45.91.116.6:4976 , piece_id=QBO6PESXU3YQNMTGK3WRI572VKQ3XE6XVJTHDKOXNI2KXU6K36PQ, seg_pos={0 0}, upload=536680]: Long tail cancellation due to optimal threshold
Dec 31 03:53:47.546876056 (612h24m47.417959101s) [noise=true , address=45.91.116.10:4083 , piece_id=LKI324QXO7PBDIPVJUXA5UKBX2OBSWF6MY6M23SAODBPUOSNZE6A, seg_pos={0 0}, upload=536680]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF
Dec 31 03:53:47.546895543 (612h24m47.417978588s) [noise=true , address=45.91.116.10:4083 , piece_id=LKI324QXO7PBDIPVJUXA5UKBX2OBSWF6MY6M23SAODBPUOSNZE6A, seg_pos={0 0}, upload=536680]: Long tail cancellation due to optimal threshold
Dec 31 03:53:47.547042914 (612h24m47.418125949s) [seg_pos={0 0}, upload=536680]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 31 03:53:47.547164576 (612h24m47.418247611s) [upload=536680]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 31 03:53:47.547173192 (612h24m47.418256227s) [upload=536680]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 31 03:53:47.838472141 (612h24m47.709555186s) [upload=536680]: Done uploading segments...
Dec 31 03:53:47.838498089 (612h24m47.709581134s) [upload=536680]: Upload finished. err: <nil>
Duration: 30.003094048s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [...................X....X.X..X..............................] duration:
13.100407039s trace:
https://jaeger.dev.storj.io/trace/5e7c34cde28aa313
FAIL (2025-12-31 03:54:00 UTC)
orbiter warning: context deadline exceeded
Duration: 13.100453218s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
14 out of 60 is failed (threshold=6) [..X.X...X...X...X.X.X...X......X..XX.....X....X............X] duration:
5.002323184s trace:
https://jaeger.dev.storj.io/trace/75a407a9ddc222e1
FAIL (2025-12-31 03:54:00 UTC)
orbiter warning: context deadline exceeded
Duration: 5.002333913s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [.....................X......................................] duration:
50.137826298s trace:
https://jaeger.dev.storj.io/trace/634c422994952cfc
FAIL (2025-12-31 03:53:59 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 121Xa3rvgKQhHc5NBufs57Ybnw3fYKUjgRyK32TTwWoi4E8m57o, Piece ID: 7ZJODSC7K4XWDIBJGMYHMGRFCXZGHTSGWTWPO7NKPWGGT73NFGOQ) context deadline exceeded; piecestore: (Node ID: 12caEiJobigXcGWwaX3Vt16zd2NXqk1WzLWxZME44E8Q8wdhn3q, Piece ID: RTZZG5ZDBPM74ZXITWZZ6LKJCKUOQ3YY4UUTMZ5VRVKFZE4YKAUA) context deadline exceeded; piecestore: (Node ID: 1zwuzCLWy2q166Ld52WUG9fvXNdJWsQrg2NearHYg7pyxpXFen, Piece ID: EJWBT3NCXTW5JSOIICAASNNDWFMIM2AO2SAE26S5POMMLEOYUH7Q) context deadline exceeded
======LOGS======
) [noise=true , address=193.149.164.143:4704 , piece_id=XIHZWOG63PEC3AQOHHWBLF53Y3B42NOO6TRNEUR3CIJB3YZTX3LQ, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.363237580 (611h51m51.234320615s) [noise=true , address=145.79.227.15:4889 , piece_id=EACS3IK7KTZI73UJEUOZB33TZN64WDWRC7TV5YJZ5ZMQVATQ3ORQ, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.364121825 (611h51m51.23520486s) [noise=true , address=193.149.164.143:4699 , piece_id=5LCUJBU4MEYEA7CQCA2FVVX6VCEX7SU3W4N3BBPQMLP6AHG5ZKKQ, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.364545251 (611h51m51.235628286s) [noise=true , address=45.91.116.11:4143 , piece_id=RTZZG5ZDBPM74ZXITWZZ6LKJCKUOQ3YY4UUTMZ5VRVKFZE4YKAUA, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.364959129 (611h51m51.236042174s) [noise=true , address=145.79.227.3:4176 , piece_id=TBPK43ULJ75RKDV5P63GXNVYZWG7TIQ3YLKMBLU55OQ2PFGSBH7A, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.365381323 (611h51m51.236464368s) [noise=true , address=145.79.227.44:4217 , piece_id=B5JSD2DUWSMSEME5CZCWFBXELK4WRMAXLLA2QQL73SB6WR6ECXUA, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.367027327 (611h51m51.238110362s) [noise=true , address=77.90.31.11:4561 , piece_id=IRE2JNK63DRH52RXBC5D3MRL6QO5S5N44UFVFWDRNY7LZBQ3IFMQ, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.367801231 (611h51m51.238884326s) [noise=true , address=145.79.227.40:4058 , piece_id=ECUZDPLHYQW3KYSTHKWBPD3CCTUCMAZ6XMOSH5S46ACF2T5366BQ, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.369086348 (611h51m51.240169404s) [noise=true , address=77.90.31.10:4536 , piece_id=EJWBT3NCXTW5JSOIICAASNNDWFMIM2AO2SAE26S5POMMLEOYUH7Q, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.370451017 (611h51m51.241534062s) [noise=true , address=77.90.31.19:4861 , piece_id=U7I2SDPMKDL4RLP2CIWZUSCEWPM5LP67VMUOLFKBHHVSCXL55MYQ, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.371980618 (611h51m51.243063654s) [noise=true , address=64.49.13.26:4955 , piece_id=53IGVDHKGGBIEKVJ6A52Q647E74KVI4FFRQU7VYK7POFU4CCX4HA, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.374009812 (611h51m51.245092847s) [noise=true , address=77.90.31.14:4671 , piece_id=2YXJ5RJERCNTD3TJHJHF4WBHR3DEOHDIOWI53H4PDS4EWQECEU2A, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.374500948 (611h51m51.245583993s) [noise=true , address=77.90.31.13:4632 , piece_id=72ADTLC2JC7Z3Z4GYKPECNAK6QMCMONGLUKZKQFLWN5UIQ4IPZYQ, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.392662589 (611h51m51.263745634s) [noise=true , address=145.79.227.32:4865 , piece_id=ZXAGYGEC35VVET2IAX2KJII4PUK3DFQXOFAYCC5R7KV5PNQCSUIA, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.393651513 (611h51m51.264734558s) [noise=true , address=145.79.227.31:4804 , piece_id=4E3PGEBPB2BFUAHNXJI4MZFWNNNZMMFDF5UBTT2UM4F2NHOHCIQQ, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.396266312 (611h51m51.267349357s) [noise=true , address=145.79.227.29:4966 , piece_id=N4JGQH23KY2BMVZZWWX7CKZN2FC2FJBFNAZEGZ7CRMRQAYEAZJLA, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:52.828565815 (611h51m52.69964886s) [upload=536207]: Next returned nil segment
Dec 31 03:20:52.828580594 (611h51m52.699663629s) [upload=536207]: Waiting for error group managing segments...
Dec 31 03:20:53.204430167 (611h51m53.075513202s) [noise=true , address=145.79.227.15:4889 , piece_id=EACS3IK7KTZI73UJEUOZB33TZN64WDWRC7TV5YJZ5ZMQVATQ3ORQ, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:53.265960889 (611h51m53.137043924s) [noise=true , address=145.79.227.31:4804 , piece_id=4E3PGEBPB2BFUAHNXJI4MZFWNNNZMMFDF5UBTT2UM4F2NHOHCIQQ, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:53.266193351 (611h51m53.137276386s) [noise=true , address=145.79.227.32:4865 , piece_id=ZXAGYGEC35VVET2IAX2KJII4PUK3DFQXOFAYCC5R7KV5PNQCSUIA, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:53.331763405 (611h51m53.20284644s) [noise=true , address=5.231.73.2:4035 , piece_id=7ZJODSC7K4XWDIBJGMYHMGRFCXZGHTSGWTWPO7NKPWGGT73NFGOQ, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:53.412602422 (611h51m53.283685467s) [noise=true , address=77.90.31.10:4536 , piece_id=EJWBT3NCXTW5JSOIICAASNNDWFMIM2AO2SAE26S5POMMLEOYUH7Q, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:53.419806240 (611h51m53.290889275s) [noise=true , address=77.90.31.14:4671 , piece_id=2YXJ5RJERCNTD3TJHJHF4WBHR3DEOHDIOWI53H4PDS4EWQECEU2A, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:53.498590626 (611h51m53.369673671s) [noise=true , address=77.90.31.11:4561 , piece_id=IRE2JNK63DRH52RXBC5D3MRL6QO5S5N44UFVFWDRNY7LZBQ3IFMQ, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:53.520983294 (611h51m53.392066339s) [noise=true , address=45.91.116.11:4143 , piece_id=RTZZG5ZDBPM74ZXITWZZ6LKJCKUOQ3YY4UUTMZ5VRVKFZE4YKAUA, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:53.585203506 (611h51m53.456286541s) [noise=true , address=77.90.31.19:4861 , piece_id=U7I2SDPMKDL4RLP2CIWZUSCEWPM5LP67VMUOLFKBHHVSCXL55MYQ, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:54.188950073 (611h51m54.060033119s) [noise=true , address=145.79.227.44:4217 , piece_id=B5JSD2DUWSMSEME5CZCWFBXELK4WRMAXLLA2QQL73SB6WR6ECXUA, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:54.204771008 (611h51m54.075854053s) [noise=true , address=145.79.227.40:4058 , piece_id=ECUZDPLHYQW3KYSTHKWBPD3CCTUCMAZ6XMOSH5S46ACF2T5366BQ, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:54.231988834 (611h51m54.103071879s) [noise=true , address=193.149.164.143:4699 , piece_id=5LCUJBU4MEYEA7CQCA2FVVX6VCEX7SU3W4N3BBPQMLP6AHG5ZKKQ, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:54.275280076 (611h51m54.146363121s) [noise=true , address=145.79.227.29:4966 , piece_id=N4JGQH23KY2BMVZZWWX7CKZN2FC2FJBFNAZEGZ7CRMRQAYEAZJLA, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:54.307143628 (611h51m54.178226673s) [noise=true , address=77.90.31.12:4602 , piece_id=ZPVIMHS5CQAKREW456SPNTHR3WM6IWZDB6AJZIYFREALJ6V63O6A, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:54.307168195 (611h51m54.17825124s) [seg_pos={0 0}, upload=536207]: Segment reached optimal threshold of 14 pieces.
Dec 31 03:20:54.307445734 (611h51m54.178528789s) [noise=true , address=77.90.31.13:4632 , piece_id=72ADTLC2JC7Z3Z4GYKPECNAK6QMCMONGLUKZKQFLWN5UIQ4IPZYQ, seg_pos={0 0}, upload=536207]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13iPed9zYrthvh36xo3k4gsqm4tz86iVkNvAr3wauuArQLquyS): protocol: expected piece hash; context canceled; EOF
Dec 31 03:20:54.307457145 (611h51m54.1785402s) [noise=true , address=77.90.31.13:4632 , piece_id=72ADTLC2JC7Z3Z4GYKPECNAK6QMCMONGLUKZKQFLWN5UIQ4IPZYQ, seg_pos={0 0}, upload=536207]: Long tail cancellation due to optimal threshold
Dec 31 03:20:54.307494436 (611h51m54.178577471s) [noise=true , address=193.149.164.143:4704 , piece_id=XIHZWOG63PEC3AQOHHWBLF53Y3B42NOO6TRNEUR3CIJB3YZTX3LQ, seg_pos={0 0}, upload=536207]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bUVmWsfAfQiPWhDkYRNGhims8TLQqGqvhrZMYugLLTdmLk2xY): protocol: expected piece hash; context canceled; EOF
Dec 31 03:20:54.307513032 (611h51m54.178596067s) [noise=true , address=193.149.164.143:4704 , piece_id=XIHZWOG63PEC3AQOHHWBLF53Y3B42NOO6TRNEUR3CIJB3YZTX3LQ, seg_pos={0 0}, upload=536207]: Long tail cancellation due to optimal threshold
Dec 31 03:20:54.307792985 (611h51m54.17887602s) [noise=true , address=145.79.227.3:4176 , piece_id=TBPK43ULJ75RKDV5P63GXNVYZWG7TIQ3YLKMBLU55OQ2PFGSBH7A, seg_pos={0 0}, upload=536207]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kGgrsNn5tv2GkcuVHHArCACwRB2qwbj3X3JmzVF4UL6dLRW1m): protocol: write tcp 188.166.225.170:39612->145.79.227.3:4176: use of closed network connection; write tcp 188.166.225.170:39612->145.79.227.3:4176: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39612->145.79.227.3:4176: use of closed network connection
Dec 31 03:20:54.307803214 (611h51m54.178886259s) [noise=true , address=145.79.227.3:4176 , piece_id=TBPK43ULJ75RKDV5P63GXNVYZWG7TIQ3YLKMBLU55OQ2PFGSBH7A, seg_pos={0 0}, upload=536207]: Long tail cancellation due to optimal threshold
Dec 31 03:20:54.307870673 (611h51m54.178953718s) [noise=true , address=64.49.13.26:4955 , piece_id=53IGVDHKGGBIEKVJ6A52Q647E74KVI4FFRQU7VYK7POFU4CCX4HA, seg_pos={0 0}, upload=536207]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125TV3v1CzV1Q9TPfyVHsn26NZ5XcHixKxA66t96iz62FF4SHAP): protocol: expected piece hash; context canceled; EOF
Dec 31 03:20:54.307882294 (611h51m54.178965339s) [noise=true , address=64.49.13.26:4955 , piece_id=53IGVDHKGGBIEKVJ6A52Q647E74KVI4FFRQU7VYK7POFU4CCX4HA, seg_pos={0 0}, upload=536207]: Long tail cancellation due to optimal threshold
Dec 31 03:20:54.307909116 (611h51m54.178992151s) [seg_pos={0 0}, upload=536207]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 31 03:20:54.308005189 (611h51m54.179088234s) [upload=536207]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 31 03:20:54.308013014 (611h51m54.179096049s) [upload=536207]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 31 03:20:54.593642638 (611h51m54.464725683s) [upload=536207]: Done uploading segments...
Dec 31 03:20:54.593663947 (611h51m54.464746983s) [upload=536207]: Upload finished. err: <nil>
Duration: 50.137858489s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
13 out of 60 is failed (threshold=6) [...X.X...X...X...X.X.X...X......X..XX.....X....X............] duration:
3.521490001s trace:
https://jaeger.dev.storj.io/trace/255fcfcc92407481
FAIL (2025-12-31 03:53:20 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.521515488s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [....................X....X.X..X.............................] duration:
6.645769354s trace:
https://jaeger.dev.storj.io/trace/5e7c34cde28aa313
FAIL (2025-12-31 03:53:05 UTC)
orbiter warning: context deadline exceeded
Duration: 6.645837044s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [......................X.....................................] duration:
25.343502217s trace:
https://jaeger.dev.storj.io/trace/634c422994952cfc
FAIL (2025-12-31 03:53:03 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 121Xa3rvgKQhHc5NBufs57Ybnw3fYKUjgRyK32TTwWoi4E8m57o, Piece ID: 7ZJODSC7K4XWDIBJGMYHMGRFCXZGHTSGWTWPO7NKPWGGT73NFGOQ) context deadline exceeded; piecestore: (Node ID: 12caEiJobigXcGWwaX3Vt16zd2NXqk1WzLWxZME44E8Q8wdhn3q, Piece ID: RTZZG5ZDBPM74ZXITWZZ6LKJCKUOQ3YY4UUTMZ5VRVKFZE4YKAUA) context deadline exceeded; piecestore: (Node ID: 1zwuzCLWy2q166Ld52WUG9fvXNdJWsQrg2NearHYg7pyxpXFen, Piece ID: EJWBT3NCXTW5JSOIICAASNNDWFMIM2AO2SAE26S5POMMLEOYUH7Q) context deadline exceeded
======LOGS======
) [noise=true , address=193.149.164.143:4704 , piece_id=XIHZWOG63PEC3AQOHHWBLF53Y3B42NOO6TRNEUR3CIJB3YZTX3LQ, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.363237580 (611h51m51.234320615s) [noise=true , address=145.79.227.15:4889 , piece_id=EACS3IK7KTZI73UJEUOZB33TZN64WDWRC7TV5YJZ5ZMQVATQ3ORQ, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.364121825 (611h51m51.23520486s) [noise=true , address=193.149.164.143:4699 , piece_id=5LCUJBU4MEYEA7CQCA2FVVX6VCEX7SU3W4N3BBPQMLP6AHG5ZKKQ, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.364545251 (611h51m51.235628286s) [noise=true , address=45.91.116.11:4143 , piece_id=RTZZG5ZDBPM74ZXITWZZ6LKJCKUOQ3YY4UUTMZ5VRVKFZE4YKAUA, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.364959129 (611h51m51.236042174s) [noise=true , address=145.79.227.3:4176 , piece_id=TBPK43ULJ75RKDV5P63GXNVYZWG7TIQ3YLKMBLU55OQ2PFGSBH7A, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.365381323 (611h51m51.236464368s) [noise=true , address=145.79.227.44:4217 , piece_id=B5JSD2DUWSMSEME5CZCWFBXELK4WRMAXLLA2QQL73SB6WR6ECXUA, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.367027327 (611h51m51.238110362s) [noise=true , address=77.90.31.11:4561 , piece_id=IRE2JNK63DRH52RXBC5D3MRL6QO5S5N44UFVFWDRNY7LZBQ3IFMQ, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.367801231 (611h51m51.238884326s) [noise=true , address=145.79.227.40:4058 , piece_id=ECUZDPLHYQW3KYSTHKWBPD3CCTUCMAZ6XMOSH5S46ACF2T5366BQ, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.369086348 (611h51m51.240169404s) [noise=true , address=77.90.31.10:4536 , piece_id=EJWBT3NCXTW5JSOIICAASNNDWFMIM2AO2SAE26S5POMMLEOYUH7Q, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.370451017 (611h51m51.241534062s) [noise=true , address=77.90.31.19:4861 , piece_id=U7I2SDPMKDL4RLP2CIWZUSCEWPM5LP67VMUOLFKBHHVSCXL55MYQ, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.371980618 (611h51m51.243063654s) [noise=true , address=64.49.13.26:4955 , piece_id=53IGVDHKGGBIEKVJ6A52Q647E74KVI4FFRQU7VYK7POFU4CCX4HA, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.374009812 (611h51m51.245092847s) [noise=true , address=77.90.31.14:4671 , piece_id=2YXJ5RJERCNTD3TJHJHF4WBHR3DEOHDIOWI53H4PDS4EWQECEU2A, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.374500948 (611h51m51.245583993s) [noise=true , address=77.90.31.13:4632 , piece_id=72ADTLC2JC7Z3Z4GYKPECNAK6QMCMONGLUKZKQFLWN5UIQ4IPZYQ, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.392662589 (611h51m51.263745634s) [noise=true , address=145.79.227.32:4865 , piece_id=ZXAGYGEC35VVET2IAX2KJII4PUK3DFQXOFAYCC5R7KV5PNQCSUIA, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.393651513 (611h51m51.264734558s) [noise=true , address=145.79.227.31:4804 , piece_id=4E3PGEBPB2BFUAHNXJI4MZFWNNNZMMFDF5UBTT2UM4F2NHOHCIQQ, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.396266312 (611h51m51.267349357s) [noise=true , address=145.79.227.29:4966 , piece_id=N4JGQH23KY2BMVZZWWX7CKZN2FC2FJBFNAZEGZ7CRMRQAYEAZJLA, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:52.828565815 (611h51m52.69964886s) [upload=536207]: Next returned nil segment
Dec 31 03:20:52.828580594 (611h51m52.699663629s) [upload=536207]: Waiting for error group managing segments...
Dec 31 03:20:53.204430167 (611h51m53.075513202s) [noise=true , address=145.79.227.15:4889 , piece_id=EACS3IK7KTZI73UJEUOZB33TZN64WDWRC7TV5YJZ5ZMQVATQ3ORQ, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:53.265960889 (611h51m53.137043924s) [noise=true , address=145.79.227.31:4804 , piece_id=4E3PGEBPB2BFUAHNXJI4MZFWNNNZMMFDF5UBTT2UM4F2NHOHCIQQ, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:53.266193351 (611h51m53.137276386s) [noise=true , address=145.79.227.32:4865 , piece_id=ZXAGYGEC35VVET2IAX2KJII4PUK3DFQXOFAYCC5R7KV5PNQCSUIA, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:53.331763405 (611h51m53.20284644s) [noise=true , address=5.231.73.2:4035 , piece_id=7ZJODSC7K4XWDIBJGMYHMGRFCXZGHTSGWTWPO7NKPWGGT73NFGOQ, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:53.412602422 (611h51m53.283685467s) [noise=true , address=77.90.31.10:4536 , piece_id=EJWBT3NCXTW5JSOIICAASNNDWFMIM2AO2SAE26S5POMMLEOYUH7Q, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:53.419806240 (611h51m53.290889275s) [noise=true , address=77.90.31.14:4671 , piece_id=2YXJ5RJERCNTD3TJHJHF4WBHR3DEOHDIOWI53H4PDS4EWQECEU2A, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:53.498590626 (611h51m53.369673671s) [noise=true , address=77.90.31.11:4561 , piece_id=IRE2JNK63DRH52RXBC5D3MRL6QO5S5N44UFVFWDRNY7LZBQ3IFMQ, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:53.520983294 (611h51m53.392066339s) [noise=true , address=45.91.116.11:4143 , piece_id=RTZZG5ZDBPM74ZXITWZZ6LKJCKUOQ3YY4UUTMZ5VRVKFZE4YKAUA, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:53.585203506 (611h51m53.456286541s) [noise=true , address=77.90.31.19:4861 , piece_id=U7I2SDPMKDL4RLP2CIWZUSCEWPM5LP67VMUOLFKBHHVSCXL55MYQ, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:54.188950073 (611h51m54.060033119s) [noise=true , address=145.79.227.44:4217 , piece_id=B5JSD2DUWSMSEME5CZCWFBXELK4WRMAXLLA2QQL73SB6WR6ECXUA, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:54.204771008 (611h51m54.075854053s) [noise=true , address=145.79.227.40:4058 , piece_id=ECUZDPLHYQW3KYSTHKWBPD3CCTUCMAZ6XMOSH5S46ACF2T5366BQ, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:54.231988834 (611h51m54.103071879s) [noise=true , address=193.149.164.143:4699 , piece_id=5LCUJBU4MEYEA7CQCA2FVVX6VCEX7SU3W4N3BBPQMLP6AHG5ZKKQ, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:54.275280076 (611h51m54.146363121s) [noise=true , address=145.79.227.29:4966 , piece_id=N4JGQH23KY2BMVZZWWX7CKZN2FC2FJBFNAZEGZ7CRMRQAYEAZJLA, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:54.307143628 (611h51m54.178226673s) [noise=true , address=77.90.31.12:4602 , piece_id=ZPVIMHS5CQAKREW456SPNTHR3WM6IWZDB6AJZIYFREALJ6V63O6A, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:54.307168195 (611h51m54.17825124s) [seg_pos={0 0}, upload=536207]: Segment reached optimal threshold of 14 pieces.
Dec 31 03:20:54.307445734 (611h51m54.178528789s) [noise=true , address=77.90.31.13:4632 , piece_id=72ADTLC2JC7Z3Z4GYKPECNAK6QMCMONGLUKZKQFLWN5UIQ4IPZYQ, seg_pos={0 0}, upload=536207]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13iPed9zYrthvh36xo3k4gsqm4tz86iVkNvAr3wauuArQLquyS): protocol: expected piece hash; context canceled; EOF
Dec 31 03:20:54.307457145 (611h51m54.1785402s) [noise=true , address=77.90.31.13:4632 , piece_id=72ADTLC2JC7Z3Z4GYKPECNAK6QMCMONGLUKZKQFLWN5UIQ4IPZYQ, seg_pos={0 0}, upload=536207]: Long tail cancellation due to optimal threshold
Dec 31 03:20:54.307494436 (611h51m54.178577471s) [noise=true , address=193.149.164.143:4704 , piece_id=XIHZWOG63PEC3AQOHHWBLF53Y3B42NOO6TRNEUR3CIJB3YZTX3LQ, seg_pos={0 0}, upload=536207]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bUVmWsfAfQiPWhDkYRNGhims8TLQqGqvhrZMYugLLTdmLk2xY): protocol: expected piece hash; context canceled; EOF
Dec 31 03:20:54.307513032 (611h51m54.178596067s) [noise=true , address=193.149.164.143:4704 , piece_id=XIHZWOG63PEC3AQOHHWBLF53Y3B42NOO6TRNEUR3CIJB3YZTX3LQ, seg_pos={0 0}, upload=536207]: Long tail cancellation due to optimal threshold
Dec 31 03:20:54.307792985 (611h51m54.17887602s) [noise=true , address=145.79.227.3:4176 , piece_id=TBPK43ULJ75RKDV5P63GXNVYZWG7TIQ3YLKMBLU55OQ2PFGSBH7A, seg_pos={0 0}, upload=536207]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kGgrsNn5tv2GkcuVHHArCACwRB2qwbj3X3JmzVF4UL6dLRW1m): protocol: write tcp 188.166.225.170:39612->145.79.227.3:4176: use of closed network connection; write tcp 188.166.225.170:39612->145.79.227.3:4176: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39612->145.79.227.3:4176: use of closed network connection
Dec 31 03:20:54.307803214 (611h51m54.178886259s) [noise=true , address=145.79.227.3:4176 , piece_id=TBPK43ULJ75RKDV5P63GXNVYZWG7TIQ3YLKMBLU55OQ2PFGSBH7A, seg_pos={0 0}, upload=536207]: Long tail cancellation due to optimal threshold
Dec 31 03:20:54.307870673 (611h51m54.178953718s) [noise=true , address=64.49.13.26:4955 , piece_id=53IGVDHKGGBIEKVJ6A52Q647E74KVI4FFRQU7VYK7POFU4CCX4HA, seg_pos={0 0}, upload=536207]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125TV3v1CzV1Q9TPfyVHsn26NZ5XcHixKxA66t96iz62FF4SHAP): protocol: expected piece hash; context canceled; EOF
Dec 31 03:20:54.307882294 (611h51m54.178965339s) [noise=true , address=64.49.13.26:4955 , piece_id=53IGVDHKGGBIEKVJ6A52Q647E74KVI4FFRQU7VYK7POFU4CCX4HA, seg_pos={0 0}, upload=536207]: Long tail cancellation due to optimal threshold
Dec 31 03:20:54.307909116 (611h51m54.178992151s) [seg_pos={0 0}, upload=536207]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 31 03:20:54.308005189 (611h51m54.179088234s) [upload=536207]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 31 03:20:54.308013014 (611h51m54.179096049s) [upload=536207]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 31 03:20:54.593642638 (611h51m54.464725683s) [upload=536207]: Done uploading segments...
Dec 31 03:20:54.593663947 (611h51m54.464746983s) [upload=536207]: Upload finished. err: <nil>
Duration: 25.343518357s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
23 out of 60 is failed (threshold=3) [XX.XX...X.X.......X....X..X..X....X.XXXXX.X.....X.X...XXXX..] duration:
6.97288853s trace:
https://jaeger.dev.storj.io/trace/3c21fc23d0c2d672
FAIL (2025-12-31 03:52:50 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, Piece ID: 3PFENWE6NPOFW2E3DXIWNSHSBWE4UHGXD3QZMI2DXVDUCY4XN7CQ) context deadline exceeded; piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: 4X2EIAJYU7IZRFPS74UMS53QFVBNEESY4AIKENYE4UEELQS2J56Q) context deadline exceeded
======LOGS======
03:50:41.435985545 (612h21m41.30706858s) [upload=536640]: Starting upload
Dec 31 03:50:41.436067522 (612h21m41.307150597s) [upload=536640]: Done starting upload
Dec 31 03:50:41.436155599 (612h21m41.307238644s) [upload=536640]: Uploading segments...
Dec 31 03:50:41.436312138 (612h21m41.307395183s) [upload=536640]: Got next segment. Inline: false
Dec 31 03:50:41.436352344 (612h21m41.307435389s) [upload=536640]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 31 03:50:41.436358445 (612h21m41.30744149s) [upload=536640]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 31 03:50:41.715136903 (612h21m41.586219938s) [seg_pos={0 0}, upload=536640]: Begin upload segment...
Dec 31 03:50:41.715196968 (612h21m41.586280003s) [seg_pos={0 0}, upload=536640]: Done begin upload segment.
Dec 31 03:50:41.715224771 (612h21m41.586307816s) [noise=true , address=45.91.116.8:4045 , piece_id=4X2EIAJYU7IZRFPS74UMS53QFVBNEESY4AIKENYE4UEELQS2J56Q, seg_pos={0 0}, upload=536640]: Uploading piece...
Dec 31 03:50:41.715314873 (612h21m41.586397938s) [noise=true , address=145.79.227.5:4223 , piece_id=EG7EF42LYP4M6N2W7TBDLMTSCMM4LY7GLZTWG6WTQSEQQ4B232MQ, seg_pos={0 0}, upload=536640]: Uploading piece...
Dec 31 03:50:41.715842938 (612h21m41.586925994s) [noise=true , address=145.79.227.38:4977 , piece_id=WENKTBI3WWFTHMSC3BB54SCZDXRN4WZ3GGN7ICFJSHRIP6P7LLIQ, seg_pos={0 0}, upload=536640]: Uploading piece...
Dec 31 03:50:41.716288075 (612h21m41.58737111s) [noise=true , address=145.79.227.27:4691 , piece_id=GG5NGKIIL7LRBGSDCTQKLYABL6EZWG2JJHGCJREWLKLD3PS4X2JQ, seg_pos={0 0}, upload=536640]: Uploading piece...
Dec 31 03:50:41.716636198 (612h21m41.587719243s) [noise=true , address=45.91.116.11:4118 , piece_id=L3UC67FNX4NE2JFCILDQSCHZYYYEDDGCSI4S3ZV47LK2S4OFBCRA, seg_pos={0 0}, upload=536640]: Uploading piece...
Dec 31 03:50:41.716907755 (612h21m41.587990801s) [noise=true , address=45.91.116.4:4937 , piece_id=7PCG4HOZZMNLW274P2DAHCMF4RBXOGS5Z5T3HPHOZRC7BLM6NVPA, seg_pos={0 0}, upload=536640]: Uploading piece...
Dec 31 03:50:41.717150177 (612h21m41.588233212s) [noise=true , address=145.79.227.10:4402 , piece_id=UYM7XNWTD7BJYJP4E4HX6MLBHJ334GF4ZLDEYXOUYTU5IR6OVD4A, seg_pos={0 0}, upload=536640]: Uploading piece...
Dec 31 03:50:41.717465487 (612h21m41.588548522s) [noise=true , address=145.79.227.35:4483 , piece_id=XGF5JFCKZEAJVWMZAAMO2RNXOH2WCUHVONLE5IN2W2G4FH2PRXWA, seg_pos={0 0}, upload=536640]: Uploading piece...
Dec 31 03:50:41.717730933 (612h21m41.588813978s) [noise=true , address=145.79.227.44:4195 , piece_id=HBQGY4HIWVP724CZFQDHIPNYKGUNEZ6X6TC5S3V52AQJWLHYPENQ, seg_pos={0 0}, upload=536640]: Uploading piece...
Dec 31 03:50:41.718220706 (612h21m41.589303751s) [noise=true , address=145.79.227.36:4518 , piece_id=A4TE6HEEAH7PAKOZ7N5T6T4PIYI4OD5DOSXOWKIBYPYZ7Q3W7GLQ, seg_pos={0 0}, upload=536640]: Uploading piece...
Dec 31 03:50:41.718817292 (612h21m41.589900337s) [noise=true , address=145.79.227.9:4366 , piece_id=CXXKEWDCARDNA7SOOTZMMXPNKRB5LS7BH2UTDRVTWB3AFUCCOBEQ, seg_pos={0 0}, upload=536640]: Uploading piece...
Dec 31 03:50:41.719156899 (612h21m41.590239934s) [noise=true , address=145.79.227.17:4970 , piece_id=DKBFKQ3ZLQERXKGJOQA4E4ZAMPTG4NHLKNBEXAIBVI4PLP5S5OIQ, seg_pos={0 0}, upload=536640]: Uploading piece...
Dec 31 03:50:41.719472820 (612h21m41.590555865s) [noise=true , address=145.79.227.7:4295 , piece_id=NG4JSF5BBAJOWVFJAGCHZ3JHFQKU2HBZ5TIITUGFXSOJRDWWNHBQ, seg_pos={0 0}, upload=536640]: Uploading piece...
Dec 31 03:50:41.719820672 (612h21m41.590903718s) [noise=true , address=45.91.116.9:4261 , piece_id=ZXFWYJM6Q5F4BF42FF2CEG4CCVG2ZBS2A4OXIOBSWXJ7QHBM34LA, seg_pos={0 0}, upload=536640]: Uploading piece...
Dec 31 03:50:41.720214071 (612h21m41.591297116s) [noise=true , address=45.91.116.13:4153 , piece_id=3PFENWE6NPOFW2E3DXIWNSHSBWE4UHGXD3QZMI2DXVDUCY4XN7CQ, seg_pos={0 0}, upload=536640]: Uploading piece...
Dec 31 03:50:41.720677564 (612h21m41.591760599s) [noise=true , address=145.79.227.23:4545 , piece_id=3CAYFUJB7QG3WGRXCPF7DAMRKA6ACSENSNW6BIH46RB3SKWNIE4Q, seg_pos={0 0}, upload=536640]: Uploading piece...
Dec 31 03:50:43.433243691 (612h21m43.304326726s) [upload=536640]: Next returned nil segment
Dec 31 03:50:43.433290170 (612h21m43.304373235s) [upload=536640]: Waiting for error group managing segments...
Dec 31 03:50:43.722364403 (612h21m43.593447438s) [noise=true , address=145.79.227.38:4977 , piece_id=WENKTBI3WWFTHMSC3BB54SCZDXRN4WZ3GGN7ICFJSHRIP6P7LLIQ, seg_pos={0 0}, upload=536640]: Done uploading piece. err: <nil>
Dec 31 03:50:43.722625329 (612h21m43.593708364s) [noise=true , address=145.79.227.27:4691 , piece_id=GG5NGKIIL7LRBGSDCTQKLYABL6EZWG2JJHGCJREWLKLD3PS4X2JQ, seg_pos={0 0}, upload=536640]: Done uploading piece. err: <nil>
Dec 31 03:50:43.730734130 (612h21m43.601817175s) [noise=true , address=145.79.227.36:4518 , piece_id=A4TE6HEEAH7PAKOZ7N5T6T4PIYI4OD5DOSXOWKIBYPYZ7Q3W7GLQ, seg_pos={0 0}, upload=536640]: Done uploading piece. err: <nil>
Dec 31 03:50:43.731522601 (612h21m43.602605636s) [noise=true , address=145.79.227.5:4223 , piece_id=EG7EF42LYP4M6N2W7TBDLMTSCMM4LY7GLZTWG6WTQSEQQ4B232MQ, seg_pos={0 0}, upload=536640]: Done uploading piece. err: <nil>
Dec 31 03:50:43.786266991 (612h21m43.657350036s) [noise=true , address=145.79.227.10:4402 , piece_id=UYM7XNWTD7BJYJP4E4HX6MLBHJ334GF4ZLDEYXOUYTU5IR6OVD4A, seg_pos={0 0}, upload=536640]: Done uploading piece. err: <nil>
Dec 31 03:50:43.787026367 (612h21m43.658109402s) [noise=true , address=145.79.227.17:4970 , piece_id=DKBFKQ3ZLQERXKGJOQA4E4ZAMPTG4NHLKNBEXAIBVI4PLP5S5OIQ, seg_pos={0 0}, upload=536640]: Done uploading piece. err: <nil>
Dec 31 03:50:43.788224478 (612h21m43.659307523s) [noise=true , address=145.79.227.7:4295 , piece_id=NG4JSF5BBAJOWVFJAGCHZ3JHFQKU2HBZ5TIITUGFXSOJRDWWNHBQ, seg_pos={0 0}, upload=536640]: Done uploading piece. err: <nil>
Dec 31 03:50:43.790012724 (612h21m43.661095759s) [noise=true , address=145.79.227.9:4366 , piece_id=CXXKEWDCARDNA7SOOTZMMXPNKRB5LS7BH2UTDRVTWB3AFUCCOBEQ, seg_pos={0 0}, upload=536640]: Done uploading piece. err: <nil>
Dec 31 03:50:43.790126190 (612h21m43.661209265s) [noise=true , address=145.79.227.23:4545 , piece_id=3CAYFUJB7QG3WGRXCPF7DAMRKA6ACSENSNW6BIH46RB3SKWNIE4Q, seg_pos={0 0}, upload=536640]: Done uploading piece. err: <nil>
Dec 31 03:50:44.130322174 (612h21m44.001405209s) [noise=true , address=45.91.116.4:4937 , piece_id=7PCG4HOZZMNLW274P2DAHCMF4RBXOGS5Z5T3HPHOZRC7BLM6NVPA, seg_pos={0 0}, upload=536640]: Done uploading piece. err: <nil>
Dec 31 03:50:44.221681304 (612h21m44.092764339s) [noise=true , address=45.91.116.8:4045 , piece_id=4X2EIAJYU7IZRFPS74UMS53QFVBNEESY4AIKENYE4UEELQS2J56Q, seg_pos={0 0}, upload=536640]: Done uploading piece. err: <nil>
Dec 31 03:50:44.320621638 (612h21m44.191704673s) [noise=true , address=45.91.116.9:4261 , piece_id=ZXFWYJM6Q5F4BF42FF2CEG4CCVG2ZBS2A4OXIOBSWXJ7QHBM34LA, seg_pos={0 0}, upload=536640]: Done uploading piece. err: <nil>
Dec 31 03:50:44.404346771 (612h21m44.275429806s) [noise=true , address=45.91.116.13:4153 , piece_id=3PFENWE6NPOFW2E3DXIWNSHSBWE4UHGXD3QZMI2DXVDUCY4XN7CQ, seg_pos={0 0}, upload=536640]: Done uploading piece. err: <nil>
Dec 31 03:50:44.404367130 (612h21m44.275450165s) [seg_pos={0 0}, upload=536640]: Segment reached optimal threshold of 13 pieces.
Dec 31 03:50:44.404549357 (612h21m44.275632402s) [noise=true , address=45.91.116.11:4118 , piece_id=L3UC67FNX4NE2JFCILDQSCHZYYYEDDGCSI4S3ZV47LK2S4OFBCRA, seg_pos={0 0}, upload=536640]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p): protocol: write tcp 188.166.225.170:46096->45.91.116.11:4118: use of closed network connection; write tcp 188.166.225.170:46096->45.91.116.11:4118: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46096->45.91.116.11:4118: use of closed network connection
Dec 31 03:50:44.404558765 (612h21m44.2756418s) [noise=true , address=45.91.116.11:4118 , piece_id=L3UC67FNX4NE2JFCILDQSCHZYYYEDDGCSI4S3ZV47LK2S4OFBCRA, seg_pos={0 0}, upload=536640]: Long tail cancellation due to optimal threshold
Dec 31 03:50:44.404614231 (612h21m44.275697266s) [noise=true , address=145.79.227.35:4483 , piece_id=XGF5JFCKZEAJVWMZAAMO2RNXOH2WCUHVONLE5IN2W2G4FH2PRXWA, seg_pos={0 0}, upload=536640]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1igSs2ZGX3zWVsSdmm1pnwYd7JckcyBubTuNU1gJDxQsZ9nKaY): protocol: expected piece hash; context canceled; EOF
Dec 31 03:50:44.404624460 (612h21m44.275707505s) [noise=true , address=145.79.227.35:4483 , piece_id=XGF5JFCKZEAJVWMZAAMO2RNXOH2WCUHVONLE5IN2W2G4FH2PRXWA, seg_pos={0 0}, upload=536640]: Long tail cancellation due to optimal threshold
Dec 31 03:50:44.404769556 (612h21m44.275852601s) [noise=true , address=145.79.227.44:4195 , piece_id=HBQGY4HIWVP724CZFQDHIPNYKGUNEZ6X6TC5S3V52AQJWLHYPENQ, seg_pos={0 0}, upload=536640]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa): protocol: expected piece hash; context canceled; EOF
Dec 31 03:50:44.404777681 (612h21m44.275860726s) [noise=true , address=145.79.227.44:4195 , piece_id=HBQGY4HIWVP724CZFQDHIPNYKGUNEZ6X6TC5S3V52AQJWLHYPENQ, seg_pos={0 0}, upload=536640]: Long tail cancellation due to optimal threshold
Dec 31 03:50:44.404793121 (612h21m44.275876166s) [seg_pos={0 0}, upload=536640]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 31 03:50:44.404885617 (612h21m44.275968662s) [upload=536640]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 31 03:50:44.404890737 (612h21m44.275973782s) [upload=536640]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 31 03:50:44.699143851 (612h21m44.570226886s) [upload=536640]: Done uploading segments...
Dec 31 03:50:44.699163148 (612h21m44.570246183s) [upload=536640]: Upload finished. err: <nil>
Duration: 6.97299384s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [.......................X....................................] duration:
53.108943488s trace:
https://jaeger.dev.storj.io/trace/634c422994952cfc
FAIL (2025-12-31 03:52:38 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 121Xa3rvgKQhHc5NBufs57Ybnw3fYKUjgRyK32TTwWoi4E8m57o, Piece ID: 7ZJODSC7K4XWDIBJGMYHMGRFCXZGHTSGWTWPO7NKPWGGT73NFGOQ) context deadline exceeded; piecestore: (Node ID: 12caEiJobigXcGWwaX3Vt16zd2NXqk1WzLWxZME44E8Q8wdhn3q, Piece ID: RTZZG5ZDBPM74ZXITWZZ6LKJCKUOQ3YY4UUTMZ5VRVKFZE4YKAUA) context deadline exceeded; piecestore: (Node ID: 1zwuzCLWy2q166Ld52WUG9fvXNdJWsQrg2NearHYg7pyxpXFen, Piece ID: EJWBT3NCXTW5JSOIICAASNNDWFMIM2AO2SAE26S5POMMLEOYUH7Q) context deadline exceeded
======LOGS======
) [noise=true , address=193.149.164.143:4704 , piece_id=XIHZWOG63PEC3AQOHHWBLF53Y3B42NOO6TRNEUR3CIJB3YZTX3LQ, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.363237580 (611h51m51.234320615s) [noise=true , address=145.79.227.15:4889 , piece_id=EACS3IK7KTZI73UJEUOZB33TZN64WDWRC7TV5YJZ5ZMQVATQ3ORQ, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.364121825 (611h51m51.23520486s) [noise=true , address=193.149.164.143:4699 , piece_id=5LCUJBU4MEYEA7CQCA2FVVX6VCEX7SU3W4N3BBPQMLP6AHG5ZKKQ, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.364545251 (611h51m51.235628286s) [noise=true , address=45.91.116.11:4143 , piece_id=RTZZG5ZDBPM74ZXITWZZ6LKJCKUOQ3YY4UUTMZ5VRVKFZE4YKAUA, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.364959129 (611h51m51.236042174s) [noise=true , address=145.79.227.3:4176 , piece_id=TBPK43ULJ75RKDV5P63GXNVYZWG7TIQ3YLKMBLU55OQ2PFGSBH7A, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.365381323 (611h51m51.236464368s) [noise=true , address=145.79.227.44:4217 , piece_id=B5JSD2DUWSMSEME5CZCWFBXELK4WRMAXLLA2QQL73SB6WR6ECXUA, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.367027327 (611h51m51.238110362s) [noise=true , address=77.90.31.11:4561 , piece_id=IRE2JNK63DRH52RXBC5D3MRL6QO5S5N44UFVFWDRNY7LZBQ3IFMQ, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.367801231 (611h51m51.238884326s) [noise=true , address=145.79.227.40:4058 , piece_id=ECUZDPLHYQW3KYSTHKWBPD3CCTUCMAZ6XMOSH5S46ACF2T5366BQ, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.369086348 (611h51m51.240169404s) [noise=true , address=77.90.31.10:4536 , piece_id=EJWBT3NCXTW5JSOIICAASNNDWFMIM2AO2SAE26S5POMMLEOYUH7Q, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.370451017 (611h51m51.241534062s) [noise=true , address=77.90.31.19:4861 , piece_id=U7I2SDPMKDL4RLP2CIWZUSCEWPM5LP67VMUOLFKBHHVSCXL55MYQ, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.371980618 (611h51m51.243063654s) [noise=true , address=64.49.13.26:4955 , piece_id=53IGVDHKGGBIEKVJ6A52Q647E74KVI4FFRQU7VYK7POFU4CCX4HA, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.374009812 (611h51m51.245092847s) [noise=true , address=77.90.31.14:4671 , piece_id=2YXJ5RJERCNTD3TJHJHF4WBHR3DEOHDIOWI53H4PDS4EWQECEU2A, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.374500948 (611h51m51.245583993s) [noise=true , address=77.90.31.13:4632 , piece_id=72ADTLC2JC7Z3Z4GYKPECNAK6QMCMONGLUKZKQFLWN5UIQ4IPZYQ, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.392662589 (611h51m51.263745634s) [noise=true , address=145.79.227.32:4865 , piece_id=ZXAGYGEC35VVET2IAX2KJII4PUK3DFQXOFAYCC5R7KV5PNQCSUIA, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.393651513 (611h51m51.264734558s) [noise=true , address=145.79.227.31:4804 , piece_id=4E3PGEBPB2BFUAHNXJI4MZFWNNNZMMFDF5UBTT2UM4F2NHOHCIQQ, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.396266312 (611h51m51.267349357s) [noise=true , address=145.79.227.29:4966 , piece_id=N4JGQH23KY2BMVZZWWX7CKZN2FC2FJBFNAZEGZ7CRMRQAYEAZJLA, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:52.828565815 (611h51m52.69964886s) [upload=536207]: Next returned nil segment
Dec 31 03:20:52.828580594 (611h51m52.699663629s) [upload=536207]: Waiting for error group managing segments...
Dec 31 03:20:53.204430167 (611h51m53.075513202s) [noise=true , address=145.79.227.15:4889 , piece_id=EACS3IK7KTZI73UJEUOZB33TZN64WDWRC7TV5YJZ5ZMQVATQ3ORQ, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:53.265960889 (611h51m53.137043924s) [noise=true , address=145.79.227.31:4804 , piece_id=4E3PGEBPB2BFUAHNXJI4MZFWNNNZMMFDF5UBTT2UM4F2NHOHCIQQ, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:53.266193351 (611h51m53.137276386s) [noise=true , address=145.79.227.32:4865 , piece_id=ZXAGYGEC35VVET2IAX2KJII4PUK3DFQXOFAYCC5R7KV5PNQCSUIA, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:53.331763405 (611h51m53.20284644s) [noise=true , address=5.231.73.2:4035 , piece_id=7ZJODSC7K4XWDIBJGMYHMGRFCXZGHTSGWTWPO7NKPWGGT73NFGOQ, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:53.412602422 (611h51m53.283685467s) [noise=true , address=77.90.31.10:4536 , piece_id=EJWBT3NCXTW5JSOIICAASNNDWFMIM2AO2SAE26S5POMMLEOYUH7Q, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:53.419806240 (611h51m53.290889275s) [noise=true , address=77.90.31.14:4671 , piece_id=2YXJ5RJERCNTD3TJHJHF4WBHR3DEOHDIOWI53H4PDS4EWQECEU2A, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:53.498590626 (611h51m53.369673671s) [noise=true , address=77.90.31.11:4561 , piece_id=IRE2JNK63DRH52RXBC5D3MRL6QO5S5N44UFVFWDRNY7LZBQ3IFMQ, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:53.520983294 (611h51m53.392066339s) [noise=true , address=45.91.116.11:4143 , piece_id=RTZZG5ZDBPM74ZXITWZZ6LKJCKUOQ3YY4UUTMZ5VRVKFZE4YKAUA, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:53.585203506 (611h51m53.456286541s) [noise=true , address=77.90.31.19:4861 , piece_id=U7I2SDPMKDL4RLP2CIWZUSCEWPM5LP67VMUOLFKBHHVSCXL55MYQ, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:54.188950073 (611h51m54.060033119s) [noise=true , address=145.79.227.44:4217 , piece_id=B5JSD2DUWSMSEME5CZCWFBXELK4WRMAXLLA2QQL73SB6WR6ECXUA, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:54.204771008 (611h51m54.075854053s) [noise=true , address=145.79.227.40:4058 , piece_id=ECUZDPLHYQW3KYSTHKWBPD3CCTUCMAZ6XMOSH5S46ACF2T5366BQ, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:54.231988834 (611h51m54.103071879s) [noise=true , address=193.149.164.143:4699 , piece_id=5LCUJBU4MEYEA7CQCA2FVVX6VCEX7SU3W4N3BBPQMLP6AHG5ZKKQ, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:54.275280076 (611h51m54.146363121s) [noise=true , address=145.79.227.29:4966 , piece_id=N4JGQH23KY2BMVZZWWX7CKZN2FC2FJBFNAZEGZ7CRMRQAYEAZJLA, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:54.307143628 (611h51m54.178226673s) [noise=true , address=77.90.31.12:4602 , piece_id=ZPVIMHS5CQAKREW456SPNTHR3WM6IWZDB6AJZIYFREALJ6V63O6A, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:54.307168195 (611h51m54.17825124s) [seg_pos={0 0}, upload=536207]: Segment reached optimal threshold of 14 pieces.
Dec 31 03:20:54.307445734 (611h51m54.178528789s) [noise=true , address=77.90.31.13:4632 , piece_id=72ADTLC2JC7Z3Z4GYKPECNAK6QMCMONGLUKZKQFLWN5UIQ4IPZYQ, seg_pos={0 0}, upload=536207]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13iPed9zYrthvh36xo3k4gsqm4tz86iVkNvAr3wauuArQLquyS): protocol: expected piece hash; context canceled; EOF
Dec 31 03:20:54.307457145 (611h51m54.1785402s) [noise=true , address=77.90.31.13:4632 , piece_id=72ADTLC2JC7Z3Z4GYKPECNAK6QMCMONGLUKZKQFLWN5UIQ4IPZYQ, seg_pos={0 0}, upload=536207]: Long tail cancellation due to optimal threshold
Dec 31 03:20:54.307494436 (611h51m54.178577471s) [noise=true , address=193.149.164.143:4704 , piece_id=XIHZWOG63PEC3AQOHHWBLF53Y3B42NOO6TRNEUR3CIJB3YZTX3LQ, seg_pos={0 0}, upload=536207]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bUVmWsfAfQiPWhDkYRNGhims8TLQqGqvhrZMYugLLTdmLk2xY): protocol: expected piece hash; context canceled; EOF
Dec 31 03:20:54.307513032 (611h51m54.178596067s) [noise=true , address=193.149.164.143:4704 , piece_id=XIHZWOG63PEC3AQOHHWBLF53Y3B42NOO6TRNEUR3CIJB3YZTX3LQ, seg_pos={0 0}, upload=536207]: Long tail cancellation due to optimal threshold
Dec 31 03:20:54.307792985 (611h51m54.17887602s) [noise=true , address=145.79.227.3:4176 , piece_id=TBPK43ULJ75RKDV5P63GXNVYZWG7TIQ3YLKMBLU55OQ2PFGSBH7A, seg_pos={0 0}, upload=536207]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kGgrsNn5tv2GkcuVHHArCACwRB2qwbj3X3JmzVF4UL6dLRW1m): protocol: write tcp 188.166.225.170:39612->145.79.227.3:4176: use of closed network connection; write tcp 188.166.225.170:39612->145.79.227.3:4176: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39612->145.79.227.3:4176: use of closed network connection
Dec 31 03:20:54.307803214 (611h51m54.178886259s) [noise=true , address=145.79.227.3:4176 , piece_id=TBPK43ULJ75RKDV5P63GXNVYZWG7TIQ3YLKMBLU55OQ2PFGSBH7A, seg_pos={0 0}, upload=536207]: Long tail cancellation due to optimal threshold
Dec 31 03:20:54.307870673 (611h51m54.178953718s) [noise=true , address=64.49.13.26:4955 , piece_id=53IGVDHKGGBIEKVJ6A52Q647E74KVI4FFRQU7VYK7POFU4CCX4HA, seg_pos={0 0}, upload=536207]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125TV3v1CzV1Q9TPfyVHsn26NZ5XcHixKxA66t96iz62FF4SHAP): protocol: expected piece hash; context canceled; EOF
Dec 31 03:20:54.307882294 (611h51m54.178965339s) [noise=true , address=64.49.13.26:4955 , piece_id=53IGVDHKGGBIEKVJ6A52Q647E74KVI4FFRQU7VYK7POFU4CCX4HA, seg_pos={0 0}, upload=536207]: Long tail cancellation due to optimal threshold
Dec 31 03:20:54.307909116 (611h51m54.178992151s) [seg_pos={0 0}, upload=536207]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 31 03:20:54.308005189 (611h51m54.179088234s) [upload=536207]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 31 03:20:54.308013014 (611h51m54.179096049s) [upload=536207]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 31 03:20:54.593642638 (611h51m54.464725683s) [upload=536207]: Done uploading segments...
Dec 31 03:20:54.593663947 (611h51m54.464746983s) [upload=536207]: Upload finished. err: <nil>
Duration: 53.108956644s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [.....................X....X.X..X............................] duration:
12.802537495s trace:
https://jaeger.dev.storj.io/trace/5e7c34cde28aa313
FAIL (2025-12-31 03:52:36 UTC)
orbiter warning: context deadline exceeded
Duration: 12.802561422s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
13 out of 60 is failed (threshold=6) [....X.X...X...X...X.X.X...X......X..XX.....X....X...........] duration:
3.614922409s trace:
https://jaeger.dev.storj.io/trace/255fcfcc92407481
FAIL (2025-12-31 03:52:22 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.614971613s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [......................X....X.X..X...........................] duration:
8.598491411s trace:
https://jaeger.dev.storj.io/trace/5e7c34cde28aa313
FAIL (2025-12-31 03:52:17 UTC)
orbiter warning: context deadline exceeded
Duration: 8.598598204s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
13 out of 60 is failed (threshold=6) [.....X.X...X...X...X.X.X...X......X..XX.....X....X..........] duration:
3.757544426s trace:
https://jaeger.dev.storj.io/trace/255fcfcc92407481
FAIL (2025-12-31 03:51:55 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.757569094s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
23 out of 60 is failed (threshold=3) [.XX.XX...X.X.......X....X..X..X....X.XXXXX.X.....X.X...XXXX.] duration:
5.791456569s trace:
https://jaeger.dev.storj.io/trace/3c21fc23d0c2d672
FAIL (2025-12-31 03:51:50 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, Piece ID: 3PFENWE6NPOFW2E3DXIWNSHSBWE4UHGXD3QZMI2DXVDUCY4XN7CQ) context deadline exceeded; piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: 4X2EIAJYU7IZRFPS74UMS53QFVBNEESY4AIKENYE4UEELQS2J56Q) context deadline exceeded
======LOGS======
03:50:41.435985545 (612h21m41.30706858s) [upload=536640]: Starting upload
Dec 31 03:50:41.436067522 (612h21m41.307150597s) [upload=536640]: Done starting upload
Dec 31 03:50:41.436155599 (612h21m41.307238644s) [upload=536640]: Uploading segments...
Dec 31 03:50:41.436312138 (612h21m41.307395183s) [upload=536640]: Got next segment. Inline: false
Dec 31 03:50:41.436352344 (612h21m41.307435389s) [upload=536640]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 31 03:50:41.436358445 (612h21m41.30744149s) [upload=536640]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 31 03:50:41.715136903 (612h21m41.586219938s) [seg_pos={0 0}, upload=536640]: Begin upload segment...
Dec 31 03:50:41.715196968 (612h21m41.586280003s) [seg_pos={0 0}, upload=536640]: Done begin upload segment.
Dec 31 03:50:41.715224771 (612h21m41.586307816s) [noise=true , address=45.91.116.8:4045 , piece_id=4X2EIAJYU7IZRFPS74UMS53QFVBNEESY4AIKENYE4UEELQS2J56Q, seg_pos={0 0}, upload=536640]: Uploading piece...
Dec 31 03:50:41.715314873 (612h21m41.586397938s) [noise=true , address=145.79.227.5:4223 , piece_id=EG7EF42LYP4M6N2W7TBDLMTSCMM4LY7GLZTWG6WTQSEQQ4B232MQ, seg_pos={0 0}, upload=536640]: Uploading piece...
Dec 31 03:50:41.715842938 (612h21m41.586925994s) [noise=true , address=145.79.227.38:4977 , piece_id=WENKTBI3WWFTHMSC3BB54SCZDXRN4WZ3GGN7ICFJSHRIP6P7LLIQ, seg_pos={0 0}, upload=536640]: Uploading piece...
Dec 31 03:50:41.716288075 (612h21m41.58737111s) [noise=true , address=145.79.227.27:4691 , piece_id=GG5NGKIIL7LRBGSDCTQKLYABL6EZWG2JJHGCJREWLKLD3PS4X2JQ, seg_pos={0 0}, upload=536640]: Uploading piece...
Dec 31 03:50:41.716636198 (612h21m41.587719243s) [noise=true , address=45.91.116.11:4118 , piece_id=L3UC67FNX4NE2JFCILDQSCHZYYYEDDGCSI4S3ZV47LK2S4OFBCRA, seg_pos={0 0}, upload=536640]: Uploading piece...
Dec 31 03:50:41.716907755 (612h21m41.587990801s) [noise=true , address=45.91.116.4:4937 , piece_id=7PCG4HOZZMNLW274P2DAHCMF4RBXOGS5Z5T3HPHOZRC7BLM6NVPA, seg_pos={0 0}, upload=536640]: Uploading piece...
Dec 31 03:50:41.717150177 (612h21m41.588233212s) [noise=true , address=145.79.227.10:4402 , piece_id=UYM7XNWTD7BJYJP4E4HX6MLBHJ334GF4ZLDEYXOUYTU5IR6OVD4A, seg_pos={0 0}, upload=536640]: Uploading piece...
Dec 31 03:50:41.717465487 (612h21m41.588548522s) [noise=true , address=145.79.227.35:4483 , piece_id=XGF5JFCKZEAJVWMZAAMO2RNXOH2WCUHVONLE5IN2W2G4FH2PRXWA, seg_pos={0 0}, upload=536640]: Uploading piece...
Dec 31 03:50:41.717730933 (612h21m41.588813978s) [noise=true , address=145.79.227.44:4195 , piece_id=HBQGY4HIWVP724CZFQDHIPNYKGUNEZ6X6TC5S3V52AQJWLHYPENQ, seg_pos={0 0}, upload=536640]: Uploading piece...
Dec 31 03:50:41.718220706 (612h21m41.589303751s) [noise=true , address=145.79.227.36:4518 , piece_id=A4TE6HEEAH7PAKOZ7N5T6T4PIYI4OD5DOSXOWKIBYPYZ7Q3W7GLQ, seg_pos={0 0}, upload=536640]: Uploading piece...
Dec 31 03:50:41.718817292 (612h21m41.589900337s) [noise=true , address=145.79.227.9:4366 , piece_id=CXXKEWDCARDNA7SOOTZMMXPNKRB5LS7BH2UTDRVTWB3AFUCCOBEQ, seg_pos={0 0}, upload=536640]: Uploading piece...
Dec 31 03:50:41.719156899 (612h21m41.590239934s) [noise=true , address=145.79.227.17:4970 , piece_id=DKBFKQ3ZLQERXKGJOQA4E4ZAMPTG4NHLKNBEXAIBVI4PLP5S5OIQ, seg_pos={0 0}, upload=536640]: Uploading piece...
Dec 31 03:50:41.719472820 (612h21m41.590555865s) [noise=true , address=145.79.227.7:4295 , piece_id=NG4JSF5BBAJOWVFJAGCHZ3JHFQKU2HBZ5TIITUGFXSOJRDWWNHBQ, seg_pos={0 0}, upload=536640]: Uploading piece...
Dec 31 03:50:41.719820672 (612h21m41.590903718s) [noise=true , address=45.91.116.9:4261 , piece_id=ZXFWYJM6Q5F4BF42FF2CEG4CCVG2ZBS2A4OXIOBSWXJ7QHBM34LA, seg_pos={0 0}, upload=536640]: Uploading piece...
Dec 31 03:50:41.720214071 (612h21m41.591297116s) [noise=true , address=45.91.116.13:4153 , piece_id=3PFENWE6NPOFW2E3DXIWNSHSBWE4UHGXD3QZMI2DXVDUCY4XN7CQ, seg_pos={0 0}, upload=536640]: Uploading piece...
Dec 31 03:50:41.720677564 (612h21m41.591760599s) [noise=true , address=145.79.227.23:4545 , piece_id=3CAYFUJB7QG3WGRXCPF7DAMRKA6ACSENSNW6BIH46RB3SKWNIE4Q, seg_pos={0 0}, upload=536640]: Uploading piece...
Dec 31 03:50:43.433243691 (612h21m43.304326726s) [upload=536640]: Next returned nil segment
Dec 31 03:50:43.433290170 (612h21m43.304373235s) [upload=536640]: Waiting for error group managing segments...
Dec 31 03:50:43.722364403 (612h21m43.593447438s) [noise=true , address=145.79.227.38:4977 , piece_id=WENKTBI3WWFTHMSC3BB54SCZDXRN4WZ3GGN7ICFJSHRIP6P7LLIQ, seg_pos={0 0}, upload=536640]: Done uploading piece. err: <nil>
Dec 31 03:50:43.722625329 (612h21m43.593708364s) [noise=true , address=145.79.227.27:4691 , piece_id=GG5NGKIIL7LRBGSDCTQKLYABL6EZWG2JJHGCJREWLKLD3PS4X2JQ, seg_pos={0 0}, upload=536640]: Done uploading piece. err: <nil>
Dec 31 03:50:43.730734130 (612h21m43.601817175s) [noise=true , address=145.79.227.36:4518 , piece_id=A4TE6HEEAH7PAKOZ7N5T6T4PIYI4OD5DOSXOWKIBYPYZ7Q3W7GLQ, seg_pos={0 0}, upload=536640]: Done uploading piece. err: <nil>
Dec 31 03:50:43.731522601 (612h21m43.602605636s) [noise=true , address=145.79.227.5:4223 , piece_id=EG7EF42LYP4M6N2W7TBDLMTSCMM4LY7GLZTWG6WTQSEQQ4B232MQ, seg_pos={0 0}, upload=536640]: Done uploading piece. err: <nil>
Dec 31 03:50:43.786266991 (612h21m43.657350036s) [noise=true , address=145.79.227.10:4402 , piece_id=UYM7XNWTD7BJYJP4E4HX6MLBHJ334GF4ZLDEYXOUYTU5IR6OVD4A, seg_pos={0 0}, upload=536640]: Done uploading piece. err: <nil>
Dec 31 03:50:43.787026367 (612h21m43.658109402s) [noise=true , address=145.79.227.17:4970 , piece_id=DKBFKQ3ZLQERXKGJOQA4E4ZAMPTG4NHLKNBEXAIBVI4PLP5S5OIQ, seg_pos={0 0}, upload=536640]: Done uploading piece. err: <nil>
Dec 31 03:50:43.788224478 (612h21m43.659307523s) [noise=true , address=145.79.227.7:4295 , piece_id=NG4JSF5BBAJOWVFJAGCHZ3JHFQKU2HBZ5TIITUGFXSOJRDWWNHBQ, seg_pos={0 0}, upload=536640]: Done uploading piece. err: <nil>
Dec 31 03:50:43.790012724 (612h21m43.661095759s) [noise=true , address=145.79.227.9:4366 , piece_id=CXXKEWDCARDNA7SOOTZMMXPNKRB5LS7BH2UTDRVTWB3AFUCCOBEQ, seg_pos={0 0}, upload=536640]: Done uploading piece. err: <nil>
Dec 31 03:50:43.790126190 (612h21m43.661209265s) [noise=true , address=145.79.227.23:4545 , piece_id=3CAYFUJB7QG3WGRXCPF7DAMRKA6ACSENSNW6BIH46RB3SKWNIE4Q, seg_pos={0 0}, upload=536640]: Done uploading piece. err: <nil>
Dec 31 03:50:44.130322174 (612h21m44.001405209s) [noise=true , address=45.91.116.4:4937 , piece_id=7PCG4HOZZMNLW274P2DAHCMF4RBXOGS5Z5T3HPHOZRC7BLM6NVPA, seg_pos={0 0}, upload=536640]: Done uploading piece. err: <nil>
Dec 31 03:50:44.221681304 (612h21m44.092764339s) [noise=true , address=45.91.116.8:4045 , piece_id=4X2EIAJYU7IZRFPS74UMS53QFVBNEESY4AIKENYE4UEELQS2J56Q, seg_pos={0 0}, upload=536640]: Done uploading piece. err: <nil>
Dec 31 03:50:44.320621638 (612h21m44.191704673s) [noise=true , address=45.91.116.9:4261 , piece_id=ZXFWYJM6Q5F4BF42FF2CEG4CCVG2ZBS2A4OXIOBSWXJ7QHBM34LA, seg_pos={0 0}, upload=536640]: Done uploading piece. err: <nil>
Dec 31 03:50:44.404346771 (612h21m44.275429806s) [noise=true , address=45.91.116.13:4153 , piece_id=3PFENWE6NPOFW2E3DXIWNSHSBWE4UHGXD3QZMI2DXVDUCY4XN7CQ, seg_pos={0 0}, upload=536640]: Done uploading piece. err: <nil>
Dec 31 03:50:44.404367130 (612h21m44.275450165s) [seg_pos={0 0}, upload=536640]: Segment reached optimal threshold of 13 pieces.
Dec 31 03:50:44.404549357 (612h21m44.275632402s) [noise=true , address=45.91.116.11:4118 , piece_id=L3UC67FNX4NE2JFCILDQSCHZYYYEDDGCSI4S3ZV47LK2S4OFBCRA, seg_pos={0 0}, upload=536640]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p): protocol: write tcp 188.166.225.170:46096->45.91.116.11:4118: use of closed network connection; write tcp 188.166.225.170:46096->45.91.116.11:4118: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46096->45.91.116.11:4118: use of closed network connection
Dec 31 03:50:44.404558765 (612h21m44.2756418s) [noise=true , address=45.91.116.11:4118 , piece_id=L3UC67FNX4NE2JFCILDQSCHZYYYEDDGCSI4S3ZV47LK2S4OFBCRA, seg_pos={0 0}, upload=536640]: Long tail cancellation due to optimal threshold
Dec 31 03:50:44.404614231 (612h21m44.275697266s) [noise=true , address=145.79.227.35:4483 , piece_id=XGF5JFCKZEAJVWMZAAMO2RNXOH2WCUHVONLE5IN2W2G4FH2PRXWA, seg_pos={0 0}, upload=536640]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1igSs2ZGX3zWVsSdmm1pnwYd7JckcyBubTuNU1gJDxQsZ9nKaY): protocol: expected piece hash; context canceled; EOF
Dec 31 03:50:44.404624460 (612h21m44.275707505s) [noise=true , address=145.79.227.35:4483 , piece_id=XGF5JFCKZEAJVWMZAAMO2RNXOH2WCUHVONLE5IN2W2G4FH2PRXWA, seg_pos={0 0}, upload=536640]: Long tail cancellation due to optimal threshold
Dec 31 03:50:44.404769556 (612h21m44.275852601s) [noise=true , address=145.79.227.44:4195 , piece_id=HBQGY4HIWVP724CZFQDHIPNYKGUNEZ6X6TC5S3V52AQJWLHYPENQ, seg_pos={0 0}, upload=536640]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa): protocol: expected piece hash; context canceled; EOF
Dec 31 03:50:44.404777681 (612h21m44.275860726s) [noise=true , address=145.79.227.44:4195 , piece_id=HBQGY4HIWVP724CZFQDHIPNYKGUNEZ6X6TC5S3V52AQJWLHYPENQ, seg_pos={0 0}, upload=536640]: Long tail cancellation due to optimal threshold
Dec 31 03:50:44.404793121 (612h21m44.275876166s) [seg_pos={0 0}, upload=536640]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 31 03:50:44.404885617 (612h21m44.275968662s) [upload=536640]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 31 03:50:44.404890737 (612h21m44.275973782s) [upload=536640]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 31 03:50:44.699143851 (612h21m44.570226886s) [upload=536640]: Done uploading segments...
Dec 31 03:50:44.699163148 (612h21m44.570246183s) [upload=536640]: Upload finished. err: <nil>
Duration: 5.791481447s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [........................X...................................] duration:
47.203986481s trace:
https://jaeger.dev.storj.io/trace/634c422994952cfc
FAIL (2025-12-31 03:51:45 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 121Xa3rvgKQhHc5NBufs57Ybnw3fYKUjgRyK32TTwWoi4E8m57o, Piece ID: 7ZJODSC7K4XWDIBJGMYHMGRFCXZGHTSGWTWPO7NKPWGGT73NFGOQ) context deadline exceeded; piecestore: (Node ID: 12caEiJobigXcGWwaX3Vt16zd2NXqk1WzLWxZME44E8Q8wdhn3q, Piece ID: RTZZG5ZDBPM74ZXITWZZ6LKJCKUOQ3YY4UUTMZ5VRVKFZE4YKAUA) context deadline exceeded; piecestore: (Node ID: 1zwuzCLWy2q166Ld52WUG9fvXNdJWsQrg2NearHYg7pyxpXFen, Piece ID: EJWBT3NCXTW5JSOIICAASNNDWFMIM2AO2SAE26S5POMMLEOYUH7Q) context deadline exceeded
======LOGS======
) [noise=true , address=193.149.164.143:4704 , piece_id=XIHZWOG63PEC3AQOHHWBLF53Y3B42NOO6TRNEUR3CIJB3YZTX3LQ, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.363237580 (611h51m51.234320615s) [noise=true , address=145.79.227.15:4889 , piece_id=EACS3IK7KTZI73UJEUOZB33TZN64WDWRC7TV5YJZ5ZMQVATQ3ORQ, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.364121825 (611h51m51.23520486s) [noise=true , address=193.149.164.143:4699 , piece_id=5LCUJBU4MEYEA7CQCA2FVVX6VCEX7SU3W4N3BBPQMLP6AHG5ZKKQ, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.364545251 (611h51m51.235628286s) [noise=true , address=45.91.116.11:4143 , piece_id=RTZZG5ZDBPM74ZXITWZZ6LKJCKUOQ3YY4UUTMZ5VRVKFZE4YKAUA, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.364959129 (611h51m51.236042174s) [noise=true , address=145.79.227.3:4176 , piece_id=TBPK43ULJ75RKDV5P63GXNVYZWG7TIQ3YLKMBLU55OQ2PFGSBH7A, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.365381323 (611h51m51.236464368s) [noise=true , address=145.79.227.44:4217 , piece_id=B5JSD2DUWSMSEME5CZCWFBXELK4WRMAXLLA2QQL73SB6WR6ECXUA, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.367027327 (611h51m51.238110362s) [noise=true , address=77.90.31.11:4561 , piece_id=IRE2JNK63DRH52RXBC5D3MRL6QO5S5N44UFVFWDRNY7LZBQ3IFMQ, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.367801231 (611h51m51.238884326s) [noise=true , address=145.79.227.40:4058 , piece_id=ECUZDPLHYQW3KYSTHKWBPD3CCTUCMAZ6XMOSH5S46ACF2T5366BQ, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.369086348 (611h51m51.240169404s) [noise=true , address=77.90.31.10:4536 , piece_id=EJWBT3NCXTW5JSOIICAASNNDWFMIM2AO2SAE26S5POMMLEOYUH7Q, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.370451017 (611h51m51.241534062s) [noise=true , address=77.90.31.19:4861 , piece_id=U7I2SDPMKDL4RLP2CIWZUSCEWPM5LP67VMUOLFKBHHVSCXL55MYQ, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.371980618 (611h51m51.243063654s) [noise=true , address=64.49.13.26:4955 , piece_id=53IGVDHKGGBIEKVJ6A52Q647E74KVI4FFRQU7VYK7POFU4CCX4HA, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.374009812 (611h51m51.245092847s) [noise=true , address=77.90.31.14:4671 , piece_id=2YXJ5RJERCNTD3TJHJHF4WBHR3DEOHDIOWI53H4PDS4EWQECEU2A, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.374500948 (611h51m51.245583993s) [noise=true , address=77.90.31.13:4632 , piece_id=72ADTLC2JC7Z3Z4GYKPECNAK6QMCMONGLUKZKQFLWN5UIQ4IPZYQ, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.392662589 (611h51m51.263745634s) [noise=true , address=145.79.227.32:4865 , piece_id=ZXAGYGEC35VVET2IAX2KJII4PUK3DFQXOFAYCC5R7KV5PNQCSUIA, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.393651513 (611h51m51.264734558s) [noise=true , address=145.79.227.31:4804 , piece_id=4E3PGEBPB2BFUAHNXJI4MZFWNNNZMMFDF5UBTT2UM4F2NHOHCIQQ, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.396266312 (611h51m51.267349357s) [noise=true , address=145.79.227.29:4966 , piece_id=N4JGQH23KY2BMVZZWWX7CKZN2FC2FJBFNAZEGZ7CRMRQAYEAZJLA, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:52.828565815 (611h51m52.69964886s) [upload=536207]: Next returned nil segment
Dec 31 03:20:52.828580594 (611h51m52.699663629s) [upload=536207]: Waiting for error group managing segments...
Dec 31 03:20:53.204430167 (611h51m53.075513202s) [noise=true , address=145.79.227.15:4889 , piece_id=EACS3IK7KTZI73UJEUOZB33TZN64WDWRC7TV5YJZ5ZMQVATQ3ORQ, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:53.265960889 (611h51m53.137043924s) [noise=true , address=145.79.227.31:4804 , piece_id=4E3PGEBPB2BFUAHNXJI4MZFWNNNZMMFDF5UBTT2UM4F2NHOHCIQQ, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:53.266193351 (611h51m53.137276386s) [noise=true , address=145.79.227.32:4865 , piece_id=ZXAGYGEC35VVET2IAX2KJII4PUK3DFQXOFAYCC5R7KV5PNQCSUIA, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:53.331763405 (611h51m53.20284644s) [noise=true , address=5.231.73.2:4035 , piece_id=7ZJODSC7K4XWDIBJGMYHMGRFCXZGHTSGWTWPO7NKPWGGT73NFGOQ, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:53.412602422 (611h51m53.283685467s) [noise=true , address=77.90.31.10:4536 , piece_id=EJWBT3NCXTW5JSOIICAASNNDWFMIM2AO2SAE26S5POMMLEOYUH7Q, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:53.419806240 (611h51m53.290889275s) [noise=true , address=77.90.31.14:4671 , piece_id=2YXJ5RJERCNTD3TJHJHF4WBHR3DEOHDIOWI53H4PDS4EWQECEU2A, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:53.498590626 (611h51m53.369673671s) [noise=true , address=77.90.31.11:4561 , piece_id=IRE2JNK63DRH52RXBC5D3MRL6QO5S5N44UFVFWDRNY7LZBQ3IFMQ, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:53.520983294 (611h51m53.392066339s) [noise=true , address=45.91.116.11:4143 , piece_id=RTZZG5ZDBPM74ZXITWZZ6LKJCKUOQ3YY4UUTMZ5VRVKFZE4YKAUA, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:53.585203506 (611h51m53.456286541s) [noise=true , address=77.90.31.19:4861 , piece_id=U7I2SDPMKDL4RLP2CIWZUSCEWPM5LP67VMUOLFKBHHVSCXL55MYQ, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:54.188950073 (611h51m54.060033119s) [noise=true , address=145.79.227.44:4217 , piece_id=B5JSD2DUWSMSEME5CZCWFBXELK4WRMAXLLA2QQL73SB6WR6ECXUA, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:54.204771008 (611h51m54.075854053s) [noise=true , address=145.79.227.40:4058 , piece_id=ECUZDPLHYQW3KYSTHKWBPD3CCTUCMAZ6XMOSH5S46ACF2T5366BQ, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:54.231988834 (611h51m54.103071879s) [noise=true , address=193.149.164.143:4699 , piece_id=5LCUJBU4MEYEA7CQCA2FVVX6VCEX7SU3W4N3BBPQMLP6AHG5ZKKQ, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:54.275280076 (611h51m54.146363121s) [noise=true , address=145.79.227.29:4966 , piece_id=N4JGQH23KY2BMVZZWWX7CKZN2FC2FJBFNAZEGZ7CRMRQAYEAZJLA, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:54.307143628 (611h51m54.178226673s) [noise=true , address=77.90.31.12:4602 , piece_id=ZPVIMHS5CQAKREW456SPNTHR3WM6IWZDB6AJZIYFREALJ6V63O6A, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:54.307168195 (611h51m54.17825124s) [seg_pos={0 0}, upload=536207]: Segment reached optimal threshold of 14 pieces.
Dec 31 03:20:54.307445734 (611h51m54.178528789s) [noise=true , address=77.90.31.13:4632 , piece_id=72ADTLC2JC7Z3Z4GYKPECNAK6QMCMONGLUKZKQFLWN5UIQ4IPZYQ, seg_pos={0 0}, upload=536207]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13iPed9zYrthvh36xo3k4gsqm4tz86iVkNvAr3wauuArQLquyS): protocol: expected piece hash; context canceled; EOF
Dec 31 03:20:54.307457145 (611h51m54.1785402s) [noise=true , address=77.90.31.13:4632 , piece_id=72ADTLC2JC7Z3Z4GYKPECNAK6QMCMONGLUKZKQFLWN5UIQ4IPZYQ, seg_pos={0 0}, upload=536207]: Long tail cancellation due to optimal threshold
Dec 31 03:20:54.307494436 (611h51m54.178577471s) [noise=true , address=193.149.164.143:4704 , piece_id=XIHZWOG63PEC3AQOHHWBLF53Y3B42NOO6TRNEUR3CIJB3YZTX3LQ, seg_pos={0 0}, upload=536207]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bUVmWsfAfQiPWhDkYRNGhims8TLQqGqvhrZMYugLLTdmLk2xY): protocol: expected piece hash; context canceled; EOF
Dec 31 03:20:54.307513032 (611h51m54.178596067s) [noise=true , address=193.149.164.143:4704 , piece_id=XIHZWOG63PEC3AQOHHWBLF53Y3B42NOO6TRNEUR3CIJB3YZTX3LQ, seg_pos={0 0}, upload=536207]: Long tail cancellation due to optimal threshold
Dec 31 03:20:54.307792985 (611h51m54.17887602s) [noise=true , address=145.79.227.3:4176 , piece_id=TBPK43ULJ75RKDV5P63GXNVYZWG7TIQ3YLKMBLU55OQ2PFGSBH7A, seg_pos={0 0}, upload=536207]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kGgrsNn5tv2GkcuVHHArCACwRB2qwbj3X3JmzVF4UL6dLRW1m): protocol: write tcp 188.166.225.170:39612->145.79.227.3:4176: use of closed network connection; write tcp 188.166.225.170:39612->145.79.227.3:4176: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39612->145.79.227.3:4176: use of closed network connection
Dec 31 03:20:54.307803214 (611h51m54.178886259s) [noise=true , address=145.79.227.3:4176 , piece_id=TBPK43ULJ75RKDV5P63GXNVYZWG7TIQ3YLKMBLU55OQ2PFGSBH7A, seg_pos={0 0}, upload=536207]: Long tail cancellation due to optimal threshold
Dec 31 03:20:54.307870673 (611h51m54.178953718s) [noise=true , address=64.49.13.26:4955 , piece_id=53IGVDHKGGBIEKVJ6A52Q647E74KVI4FFRQU7VYK7POFU4CCX4HA, seg_pos={0 0}, upload=536207]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125TV3v1CzV1Q9TPfyVHsn26NZ5XcHixKxA66t96iz62FF4SHAP): protocol: expected piece hash; context canceled; EOF
Dec 31 03:20:54.307882294 (611h51m54.178965339s) [noise=true , address=64.49.13.26:4955 , piece_id=53IGVDHKGGBIEKVJ6A52Q647E74KVI4FFRQU7VYK7POFU4CCX4HA, seg_pos={0 0}, upload=536207]: Long tail cancellation due to optimal threshold
Dec 31 03:20:54.307909116 (611h51m54.178992151s) [seg_pos={0 0}, upload=536207]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 31 03:20:54.308005189 (611h51m54.179088234s) [upload=536207]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 31 03:20:54.308013014 (611h51m54.179096049s) [upload=536207]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 31 03:20:54.593642638 (611h51m54.464725683s) [upload=536207]: Done uploading segments...
Dec 31 03:20:54.593663947 (611h51m54.464746983s) [upload=536207]: Upload finished. err: <nil>
Duration: 47.203999705s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
13 out of 60 is failed (threshold=6) [......X.X...X...X...X.X.X...X......X..XX.....X....X.........] duration:
3.40176047s trace:
https://jaeger.dev.storj.io/trace/255fcfcc92407481
FAIL (2025-12-31 03:51:35 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.401811467s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [.......................X....X.X..X..........................] duration:
7.290212996s trace:
https://jaeger.dev.storj.io/trace/5e7c34cde28aa313
FAIL (2025-12-31 03:51:16 UTC)
orbiter warning: context deadline exceeded
Duration: 7.290326854s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
24 out of 60 is failed (threshold=3) [X.XX.XX...X.X.......X....X..X..X....X.XXXXX.X.....X.X...XXXX] duration:
30.004897897s trace:
https://jaeger.dev.storj.io/trace/3c21fc23d0c2d672
FAIL (2025-12-31 03:51:10 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, Piece ID: 3PFENWE6NPOFW2E3DXIWNSHSBWE4UHGXD3QZMI2DXVDUCY4XN7CQ) context deadline exceeded; piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: 4X2EIAJYU7IZRFPS74UMS53QFVBNEESY4AIKENYE4UEELQS2J56Q) context deadline exceeded
======LOGS======
03:50:41.435985545 (612h21m41.30706858s) [upload=536640]: Starting upload
Dec 31 03:50:41.436067522 (612h21m41.307150597s) [upload=536640]: Done starting upload
Dec 31 03:50:41.436155599 (612h21m41.307238644s) [upload=536640]: Uploading segments...
Dec 31 03:50:41.436312138 (612h21m41.307395183s) [upload=536640]: Got next segment. Inline: false
Dec 31 03:50:41.436352344 (612h21m41.307435389s) [upload=536640]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 31 03:50:41.436358445 (612h21m41.30744149s) [upload=536640]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 31 03:50:41.715136903 (612h21m41.586219938s) [seg_pos={0 0}, upload=536640]: Begin upload segment...
Dec 31 03:50:41.715196968 (612h21m41.586280003s) [seg_pos={0 0}, upload=536640]: Done begin upload segment.
Dec 31 03:50:41.715224771 (612h21m41.586307816s) [noise=true , address=45.91.116.8:4045 , piece_id=4X2EIAJYU7IZRFPS74UMS53QFVBNEESY4AIKENYE4UEELQS2J56Q, seg_pos={0 0}, upload=536640]: Uploading piece...
Dec 31 03:50:41.715314873 (612h21m41.586397938s) [noise=true , address=145.79.227.5:4223 , piece_id=EG7EF42LYP4M6N2W7TBDLMTSCMM4LY7GLZTWG6WTQSEQQ4B232MQ, seg_pos={0 0}, upload=536640]: Uploading piece...
Dec 31 03:50:41.715842938 (612h21m41.586925994s) [noise=true , address=145.79.227.38:4977 , piece_id=WENKTBI3WWFTHMSC3BB54SCZDXRN4WZ3GGN7ICFJSHRIP6P7LLIQ, seg_pos={0 0}, upload=536640]: Uploading piece...
Dec 31 03:50:41.716288075 (612h21m41.58737111s) [noise=true , address=145.79.227.27:4691 , piece_id=GG5NGKIIL7LRBGSDCTQKLYABL6EZWG2JJHGCJREWLKLD3PS4X2JQ, seg_pos={0 0}, upload=536640]: Uploading piece...
Dec 31 03:50:41.716636198 (612h21m41.587719243s) [noise=true , address=45.91.116.11:4118 , piece_id=L3UC67FNX4NE2JFCILDQSCHZYYYEDDGCSI4S3ZV47LK2S4OFBCRA, seg_pos={0 0}, upload=536640]: Uploading piece...
Dec 31 03:50:41.716907755 (612h21m41.587990801s) [noise=true , address=45.91.116.4:4937 , piece_id=7PCG4HOZZMNLW274P2DAHCMF4RBXOGS5Z5T3HPHOZRC7BLM6NVPA, seg_pos={0 0}, upload=536640]: Uploading piece...
Dec 31 03:50:41.717150177 (612h21m41.588233212s) [noise=true , address=145.79.227.10:4402 , piece_id=UYM7XNWTD7BJYJP4E4HX6MLBHJ334GF4ZLDEYXOUYTU5IR6OVD4A, seg_pos={0 0}, upload=536640]: Uploading piece...
Dec 31 03:50:41.717465487 (612h21m41.588548522s) [noise=true , address=145.79.227.35:4483 , piece_id=XGF5JFCKZEAJVWMZAAMO2RNXOH2WCUHVONLE5IN2W2G4FH2PRXWA, seg_pos={0 0}, upload=536640]: Uploading piece...
Dec 31 03:50:41.717730933 (612h21m41.588813978s) [noise=true , address=145.79.227.44:4195 , piece_id=HBQGY4HIWVP724CZFQDHIPNYKGUNEZ6X6TC5S3V52AQJWLHYPENQ, seg_pos={0 0}, upload=536640]: Uploading piece...
Dec 31 03:50:41.718220706 (612h21m41.589303751s) [noise=true , address=145.79.227.36:4518 , piece_id=A4TE6HEEAH7PAKOZ7N5T6T4PIYI4OD5DOSXOWKIBYPYZ7Q3W7GLQ, seg_pos={0 0}, upload=536640]: Uploading piece...
Dec 31 03:50:41.718817292 (612h21m41.589900337s) [noise=true , address=145.79.227.9:4366 , piece_id=CXXKEWDCARDNA7SOOTZMMXPNKRB5LS7BH2UTDRVTWB3AFUCCOBEQ, seg_pos={0 0}, upload=536640]: Uploading piece...
Dec 31 03:50:41.719156899 (612h21m41.590239934s) [noise=true , address=145.79.227.17:4970 , piece_id=DKBFKQ3ZLQERXKGJOQA4E4ZAMPTG4NHLKNBEXAIBVI4PLP5S5OIQ, seg_pos={0 0}, upload=536640]: Uploading piece...
Dec 31 03:50:41.719472820 (612h21m41.590555865s) [noise=true , address=145.79.227.7:4295 , piece_id=NG4JSF5BBAJOWVFJAGCHZ3JHFQKU2HBZ5TIITUGFXSOJRDWWNHBQ, seg_pos={0 0}, upload=536640]: Uploading piece...
Dec 31 03:50:41.719820672 (612h21m41.590903718s) [noise=true , address=45.91.116.9:4261 , piece_id=ZXFWYJM6Q5F4BF42FF2CEG4CCVG2ZBS2A4OXIOBSWXJ7QHBM34LA, seg_pos={0 0}, upload=536640]: Uploading piece...
Dec 31 03:50:41.720214071 (612h21m41.591297116s) [noise=true , address=45.91.116.13:4153 , piece_id=3PFENWE6NPOFW2E3DXIWNSHSBWE4UHGXD3QZMI2DXVDUCY4XN7CQ, seg_pos={0 0}, upload=536640]: Uploading piece...
Dec 31 03:50:41.720677564 (612h21m41.591760599s) [noise=true , address=145.79.227.23:4545 , piece_id=3CAYFUJB7QG3WGRXCPF7DAMRKA6ACSENSNW6BIH46RB3SKWNIE4Q, seg_pos={0 0}, upload=536640]: Uploading piece...
Dec 31 03:50:43.433243691 (612h21m43.304326726s) [upload=536640]: Next returned nil segment
Dec 31 03:50:43.433290170 (612h21m43.304373235s) [upload=536640]: Waiting for error group managing segments...
Dec 31 03:50:43.722364403 (612h21m43.593447438s) [noise=true , address=145.79.227.38:4977 , piece_id=WENKTBI3WWFTHMSC3BB54SCZDXRN4WZ3GGN7ICFJSHRIP6P7LLIQ, seg_pos={0 0}, upload=536640]: Done uploading piece. err: <nil>
Dec 31 03:50:43.722625329 (612h21m43.593708364s) [noise=true , address=145.79.227.27:4691 , piece_id=GG5NGKIIL7LRBGSDCTQKLYABL6EZWG2JJHGCJREWLKLD3PS4X2JQ, seg_pos={0 0}, upload=536640]: Done uploading piece. err: <nil>
Dec 31 03:50:43.730734130 (612h21m43.601817175s) [noise=true , address=145.79.227.36:4518 , piece_id=A4TE6HEEAH7PAKOZ7N5T6T4PIYI4OD5DOSXOWKIBYPYZ7Q3W7GLQ, seg_pos={0 0}, upload=536640]: Done uploading piece. err: <nil>
Dec 31 03:50:43.731522601 (612h21m43.602605636s) [noise=true , address=145.79.227.5:4223 , piece_id=EG7EF42LYP4M6N2W7TBDLMTSCMM4LY7GLZTWG6WTQSEQQ4B232MQ, seg_pos={0 0}, upload=536640]: Done uploading piece. err: <nil>
Dec 31 03:50:43.786266991 (612h21m43.657350036s) [noise=true , address=145.79.227.10:4402 , piece_id=UYM7XNWTD7BJYJP4E4HX6MLBHJ334GF4ZLDEYXOUYTU5IR6OVD4A, seg_pos={0 0}, upload=536640]: Done uploading piece. err: <nil>
Dec 31 03:50:43.787026367 (612h21m43.658109402s) [noise=true , address=145.79.227.17:4970 , piece_id=DKBFKQ3ZLQERXKGJOQA4E4ZAMPTG4NHLKNBEXAIBVI4PLP5S5OIQ, seg_pos={0 0}, upload=536640]: Done uploading piece. err: <nil>
Dec 31 03:50:43.788224478 (612h21m43.659307523s) [noise=true , address=145.79.227.7:4295 , piece_id=NG4JSF5BBAJOWVFJAGCHZ3JHFQKU2HBZ5TIITUGFXSOJRDWWNHBQ, seg_pos={0 0}, upload=536640]: Done uploading piece. err: <nil>
Dec 31 03:50:43.790012724 (612h21m43.661095759s) [noise=true , address=145.79.227.9:4366 , piece_id=CXXKEWDCARDNA7SOOTZMMXPNKRB5LS7BH2UTDRVTWB3AFUCCOBEQ, seg_pos={0 0}, upload=536640]: Done uploading piece. err: <nil>
Dec 31 03:50:43.790126190 (612h21m43.661209265s) [noise=true , address=145.79.227.23:4545 , piece_id=3CAYFUJB7QG3WGRXCPF7DAMRKA6ACSENSNW6BIH46RB3SKWNIE4Q, seg_pos={0 0}, upload=536640]: Done uploading piece. err: <nil>
Dec 31 03:50:44.130322174 (612h21m44.001405209s) [noise=true , address=45.91.116.4:4937 , piece_id=7PCG4HOZZMNLW274P2DAHCMF4RBXOGS5Z5T3HPHOZRC7BLM6NVPA, seg_pos={0 0}, upload=536640]: Done uploading piece. err: <nil>
Dec 31 03:50:44.221681304 (612h21m44.092764339s) [noise=true , address=45.91.116.8:4045 , piece_id=4X2EIAJYU7IZRFPS74UMS53QFVBNEESY4AIKENYE4UEELQS2J56Q, seg_pos={0 0}, upload=536640]: Done uploading piece. err: <nil>
Dec 31 03:50:44.320621638 (612h21m44.191704673s) [noise=true , address=45.91.116.9:4261 , piece_id=ZXFWYJM6Q5F4BF42FF2CEG4CCVG2ZBS2A4OXIOBSWXJ7QHBM34LA, seg_pos={0 0}, upload=536640]: Done uploading piece. err: <nil>
Dec 31 03:50:44.404346771 (612h21m44.275429806s) [noise=true , address=45.91.116.13:4153 , piece_id=3PFENWE6NPOFW2E3DXIWNSHSBWE4UHGXD3QZMI2DXVDUCY4XN7CQ, seg_pos={0 0}, upload=536640]: Done uploading piece. err: <nil>
Dec 31 03:50:44.404367130 (612h21m44.275450165s) [seg_pos={0 0}, upload=536640]: Segment reached optimal threshold of 13 pieces.
Dec 31 03:50:44.404549357 (612h21m44.275632402s) [noise=true , address=45.91.116.11:4118 , piece_id=L3UC67FNX4NE2JFCILDQSCHZYYYEDDGCSI4S3ZV47LK2S4OFBCRA, seg_pos={0 0}, upload=536640]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p): protocol: write tcp 188.166.225.170:46096->45.91.116.11:4118: use of closed network connection; write tcp 188.166.225.170:46096->45.91.116.11:4118: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46096->45.91.116.11:4118: use of closed network connection
Dec 31 03:50:44.404558765 (612h21m44.2756418s) [noise=true , address=45.91.116.11:4118 , piece_id=L3UC67FNX4NE2JFCILDQSCHZYYYEDDGCSI4S3ZV47LK2S4OFBCRA, seg_pos={0 0}, upload=536640]: Long tail cancellation due to optimal threshold
Dec 31 03:50:44.404614231 (612h21m44.275697266s) [noise=true , address=145.79.227.35:4483 , piece_id=XGF5JFCKZEAJVWMZAAMO2RNXOH2WCUHVONLE5IN2W2G4FH2PRXWA, seg_pos={0 0}, upload=536640]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1igSs2ZGX3zWVsSdmm1pnwYd7JckcyBubTuNU1gJDxQsZ9nKaY): protocol: expected piece hash; context canceled; EOF
Dec 31 03:50:44.404624460 (612h21m44.275707505s) [noise=true , address=145.79.227.35:4483 , piece_id=XGF5JFCKZEAJVWMZAAMO2RNXOH2WCUHVONLE5IN2W2G4FH2PRXWA, seg_pos={0 0}, upload=536640]: Long tail cancellation due to optimal threshold
Dec 31 03:50:44.404769556 (612h21m44.275852601s) [noise=true , address=145.79.227.44:4195 , piece_id=HBQGY4HIWVP724CZFQDHIPNYKGUNEZ6X6TC5S3V52AQJWLHYPENQ, seg_pos={0 0}, upload=536640]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa): protocol: expected piece hash; context canceled; EOF
Dec 31 03:50:44.404777681 (612h21m44.275860726s) [noise=true , address=145.79.227.44:4195 , piece_id=HBQGY4HIWVP724CZFQDHIPNYKGUNEZ6X6TC5S3V52AQJWLHYPENQ, seg_pos={0 0}, upload=536640]: Long tail cancellation due to optimal threshold
Dec 31 03:50:44.404793121 (612h21m44.275876166s) [seg_pos={0 0}, upload=536640]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 31 03:50:44.404885617 (612h21m44.275968662s) [upload=536640]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 31 03:50:44.404890737 (612h21m44.275973782s) [upload=536640]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 31 03:50:44.699143851 (612h21m44.570226886s) [upload=536640]: Done uploading segments...
Dec 31 03:50:44.699163148 (612h21m44.570246183s) [upload=536640]: Upload finished. err: <nil>
Duration: 30.004934768s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [.........................X..................................] duration:
39.578393117s trace:
https://jaeger.dev.storj.io/trace/634c422994952cfc
FAIL (2025-12-31 03:50:57 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 121Xa3rvgKQhHc5NBufs57Ybnw3fYKUjgRyK32TTwWoi4E8m57o, Piece ID: 7ZJODSC7K4XWDIBJGMYHMGRFCXZGHTSGWTWPO7NKPWGGT73NFGOQ) context deadline exceeded; piecestore: (Node ID: 12caEiJobigXcGWwaX3Vt16zd2NXqk1WzLWxZME44E8Q8wdhn3q, Piece ID: RTZZG5ZDBPM74ZXITWZZ6LKJCKUOQ3YY4UUTMZ5VRVKFZE4YKAUA) context deadline exceeded; piecestore: (Node ID: 1zwuzCLWy2q166Ld52WUG9fvXNdJWsQrg2NearHYg7pyxpXFen, Piece ID: EJWBT3NCXTW5JSOIICAASNNDWFMIM2AO2SAE26S5POMMLEOYUH7Q) context deadline exceeded
======LOGS======
) [noise=true , address=193.149.164.143:4704 , piece_id=XIHZWOG63PEC3AQOHHWBLF53Y3B42NOO6TRNEUR3CIJB3YZTX3LQ, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.363237580 (611h51m51.234320615s) [noise=true , address=145.79.227.15:4889 , piece_id=EACS3IK7KTZI73UJEUOZB33TZN64WDWRC7TV5YJZ5ZMQVATQ3ORQ, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.364121825 (611h51m51.23520486s) [noise=true , address=193.149.164.143:4699 , piece_id=5LCUJBU4MEYEA7CQCA2FVVX6VCEX7SU3W4N3BBPQMLP6AHG5ZKKQ, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.364545251 (611h51m51.235628286s) [noise=true , address=45.91.116.11:4143 , piece_id=RTZZG5ZDBPM74ZXITWZZ6LKJCKUOQ3YY4UUTMZ5VRVKFZE4YKAUA, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.364959129 (611h51m51.236042174s) [noise=true , address=145.79.227.3:4176 , piece_id=TBPK43ULJ75RKDV5P63GXNVYZWG7TIQ3YLKMBLU55OQ2PFGSBH7A, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.365381323 (611h51m51.236464368s) [noise=true , address=145.79.227.44:4217 , piece_id=B5JSD2DUWSMSEME5CZCWFBXELK4WRMAXLLA2QQL73SB6WR6ECXUA, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.367027327 (611h51m51.238110362s) [noise=true , address=77.90.31.11:4561 , piece_id=IRE2JNK63DRH52RXBC5D3MRL6QO5S5N44UFVFWDRNY7LZBQ3IFMQ, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.367801231 (611h51m51.238884326s) [noise=true , address=145.79.227.40:4058 , piece_id=ECUZDPLHYQW3KYSTHKWBPD3CCTUCMAZ6XMOSH5S46ACF2T5366BQ, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.369086348 (611h51m51.240169404s) [noise=true , address=77.90.31.10:4536 , piece_id=EJWBT3NCXTW5JSOIICAASNNDWFMIM2AO2SAE26S5POMMLEOYUH7Q, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.370451017 (611h51m51.241534062s) [noise=true , address=77.90.31.19:4861 , piece_id=U7I2SDPMKDL4RLP2CIWZUSCEWPM5LP67VMUOLFKBHHVSCXL55MYQ, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.371980618 (611h51m51.243063654s) [noise=true , address=64.49.13.26:4955 , piece_id=53IGVDHKGGBIEKVJ6A52Q647E74KVI4FFRQU7VYK7POFU4CCX4HA, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.374009812 (611h51m51.245092847s) [noise=true , address=77.90.31.14:4671 , piece_id=2YXJ5RJERCNTD3TJHJHF4WBHR3DEOHDIOWI53H4PDS4EWQECEU2A, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.374500948 (611h51m51.245583993s) [noise=true , address=77.90.31.13:4632 , piece_id=72ADTLC2JC7Z3Z4GYKPECNAK6QMCMONGLUKZKQFLWN5UIQ4IPZYQ, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.392662589 (611h51m51.263745634s) [noise=true , address=145.79.227.32:4865 , piece_id=ZXAGYGEC35VVET2IAX2KJII4PUK3DFQXOFAYCC5R7KV5PNQCSUIA, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.393651513 (611h51m51.264734558s) [noise=true , address=145.79.227.31:4804 , piece_id=4E3PGEBPB2BFUAHNXJI4MZFWNNNZMMFDF5UBTT2UM4F2NHOHCIQQ, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.396266312 (611h51m51.267349357s) [noise=true , address=145.79.227.29:4966 , piece_id=N4JGQH23KY2BMVZZWWX7CKZN2FC2FJBFNAZEGZ7CRMRQAYEAZJLA, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:52.828565815 (611h51m52.69964886s) [upload=536207]: Next returned nil segment
Dec 31 03:20:52.828580594 (611h51m52.699663629s) [upload=536207]: Waiting for error group managing segments...
Dec 31 03:20:53.204430167 (611h51m53.075513202s) [noise=true , address=145.79.227.15:4889 , piece_id=EACS3IK7KTZI73UJEUOZB33TZN64WDWRC7TV5YJZ5ZMQVATQ3ORQ, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:53.265960889 (611h51m53.137043924s) [noise=true , address=145.79.227.31:4804 , piece_id=4E3PGEBPB2BFUAHNXJI4MZFWNNNZMMFDF5UBTT2UM4F2NHOHCIQQ, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:53.266193351 (611h51m53.137276386s) [noise=true , address=145.79.227.32:4865 , piece_id=ZXAGYGEC35VVET2IAX2KJII4PUK3DFQXOFAYCC5R7KV5PNQCSUIA, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:53.331763405 (611h51m53.20284644s) [noise=true , address=5.231.73.2:4035 , piece_id=7ZJODSC7K4XWDIBJGMYHMGRFCXZGHTSGWTWPO7NKPWGGT73NFGOQ, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:53.412602422 (611h51m53.283685467s) [noise=true , address=77.90.31.10:4536 , piece_id=EJWBT3NCXTW5JSOIICAASNNDWFMIM2AO2SAE26S5POMMLEOYUH7Q, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:53.419806240 (611h51m53.290889275s) [noise=true , address=77.90.31.14:4671 , piece_id=2YXJ5RJERCNTD3TJHJHF4WBHR3DEOHDIOWI53H4PDS4EWQECEU2A, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:53.498590626 (611h51m53.369673671s) [noise=true , address=77.90.31.11:4561 , piece_id=IRE2JNK63DRH52RXBC5D3MRL6QO5S5N44UFVFWDRNY7LZBQ3IFMQ, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:53.520983294 (611h51m53.392066339s) [noise=true , address=45.91.116.11:4143 , piece_id=RTZZG5ZDBPM74ZXITWZZ6LKJCKUOQ3YY4UUTMZ5VRVKFZE4YKAUA, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:53.585203506 (611h51m53.456286541s) [noise=true , address=77.90.31.19:4861 , piece_id=U7I2SDPMKDL4RLP2CIWZUSCEWPM5LP67VMUOLFKBHHVSCXL55MYQ, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:54.188950073 (611h51m54.060033119s) [noise=true , address=145.79.227.44:4217 , piece_id=B5JSD2DUWSMSEME5CZCWFBXELK4WRMAXLLA2QQL73SB6WR6ECXUA, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:54.204771008 (611h51m54.075854053s) [noise=true , address=145.79.227.40:4058 , piece_id=ECUZDPLHYQW3KYSTHKWBPD3CCTUCMAZ6XMOSH5S46ACF2T5366BQ, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:54.231988834 (611h51m54.103071879s) [noise=true , address=193.149.164.143:4699 , piece_id=5LCUJBU4MEYEA7CQCA2FVVX6VCEX7SU3W4N3BBPQMLP6AHG5ZKKQ, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:54.275280076 (611h51m54.146363121s) [noise=true , address=145.79.227.29:4966 , piece_id=N4JGQH23KY2BMVZZWWX7CKZN2FC2FJBFNAZEGZ7CRMRQAYEAZJLA, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:54.307143628 (611h51m54.178226673s) [noise=true , address=77.90.31.12:4602 , piece_id=ZPVIMHS5CQAKREW456SPNTHR3WM6IWZDB6AJZIYFREALJ6V63O6A, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:54.307168195 (611h51m54.17825124s) [seg_pos={0 0}, upload=536207]: Segment reached optimal threshold of 14 pieces.
Dec 31 03:20:54.307445734 (611h51m54.178528789s) [noise=true , address=77.90.31.13:4632 , piece_id=72ADTLC2JC7Z3Z4GYKPECNAK6QMCMONGLUKZKQFLWN5UIQ4IPZYQ, seg_pos={0 0}, upload=536207]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13iPed9zYrthvh36xo3k4gsqm4tz86iVkNvAr3wauuArQLquyS): protocol: expected piece hash; context canceled; EOF
Dec 31 03:20:54.307457145 (611h51m54.1785402s) [noise=true , address=77.90.31.13:4632 , piece_id=72ADTLC2JC7Z3Z4GYKPECNAK6QMCMONGLUKZKQFLWN5UIQ4IPZYQ, seg_pos={0 0}, upload=536207]: Long tail cancellation due to optimal threshold
Dec 31 03:20:54.307494436 (611h51m54.178577471s) [noise=true , address=193.149.164.143:4704 , piece_id=XIHZWOG63PEC3AQOHHWBLF53Y3B42NOO6TRNEUR3CIJB3YZTX3LQ, seg_pos={0 0}, upload=536207]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bUVmWsfAfQiPWhDkYRNGhims8TLQqGqvhrZMYugLLTdmLk2xY): protocol: expected piece hash; context canceled; EOF
Dec 31 03:20:54.307513032 (611h51m54.178596067s) [noise=true , address=193.149.164.143:4704 , piece_id=XIHZWOG63PEC3AQOHHWBLF53Y3B42NOO6TRNEUR3CIJB3YZTX3LQ, seg_pos={0 0}, upload=536207]: Long tail cancellation due to optimal threshold
Dec 31 03:20:54.307792985 (611h51m54.17887602s) [noise=true , address=145.79.227.3:4176 , piece_id=TBPK43ULJ75RKDV5P63GXNVYZWG7TIQ3YLKMBLU55OQ2PFGSBH7A, seg_pos={0 0}, upload=536207]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kGgrsNn5tv2GkcuVHHArCACwRB2qwbj3X3JmzVF4UL6dLRW1m): protocol: write tcp 188.166.225.170:39612->145.79.227.3:4176: use of closed network connection; write tcp 188.166.225.170:39612->145.79.227.3:4176: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39612->145.79.227.3:4176: use of closed network connection
Dec 31 03:20:54.307803214 (611h51m54.178886259s) [noise=true , address=145.79.227.3:4176 , piece_id=TBPK43ULJ75RKDV5P63GXNVYZWG7TIQ3YLKMBLU55OQ2PFGSBH7A, seg_pos={0 0}, upload=536207]: Long tail cancellation due to optimal threshold
Dec 31 03:20:54.307870673 (611h51m54.178953718s) [noise=true , address=64.49.13.26:4955 , piece_id=53IGVDHKGGBIEKVJ6A52Q647E74KVI4FFRQU7VYK7POFU4CCX4HA, seg_pos={0 0}, upload=536207]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125TV3v1CzV1Q9TPfyVHsn26NZ5XcHixKxA66t96iz62FF4SHAP): protocol: expected piece hash; context canceled; EOF
Dec 31 03:20:54.307882294 (611h51m54.178965339s) [noise=true , address=64.49.13.26:4955 , piece_id=53IGVDHKGGBIEKVJ6A52Q647E74KVI4FFRQU7VYK7POFU4CCX4HA, seg_pos={0 0}, upload=536207]: Long tail cancellation due to optimal threshold
Dec 31 03:20:54.307909116 (611h51m54.178992151s) [seg_pos={0 0}, upload=536207]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 31 03:20:54.308005189 (611h51m54.179088234s) [upload=536207]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 31 03:20:54.308013014 (611h51m54.179096049s) [upload=536207]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 31 03:20:54.593642638 (611h51m54.464725683s) [upload=536207]: Done uploading segments...
Dec 31 03:20:54.593663947 (611h51m54.464746983s) [upload=536207]: Upload finished. err: <nil>
Duration: 39.578412636s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
23 out of 60 is failed (threshold=3) [.X.XX.XX...X.X.......X....X..X..X....X.XXXXX.X.....X.X...XXX] duration:
30.003459619s trace:
https://jaeger.dev.storj.io/trace/22c9f8ea81f557dc
FAIL (2025-12-31 03:50:40 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 31 03:50:11.196423436 (612h21m11.067506471s) [upload=536634]: Starting upload
Dec 31 03:50:11.196483481 (612h21m11.067566526s) [upload=536634]: Done starting upload
Dec 31 03:50:11.196606214 (612h21m11.067689259s) [upload=536634]: Uploading segments...
Dec 31 03:50:11.196769055 (612h21m11.0678521s) [upload=536634]: Got next segment. Inline: false
Dec 31 03:50:11.196813679 (612h21m11.067896724s) [upload=536634]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 31 03:50:11.196820973 (612h21m11.067904008s) [upload=536634]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 31 03:50:11.444723089 (612h21m11.315806134s) [seg_pos={0 0}, upload=536634]: Begin upload segment...
Dec 31 03:50:11.444784415 (612h21m11.31586745s) [seg_pos={0 0}, upload=536634]: Done begin upload segment.
Dec 31 03:50:11.444820373 (612h21m11.315903419s) [noise=true , address=45.91.116.8:4045 , piece_id=CVUQDF5JIRWOUW57SLUOKGL4EFKN4JVSLOK5YLK6V3DQD3VZXXUQ, seg_pos={0 0}, upload=536634]: Uploading piece...
Dec 31 03:50:11.444932267 (612h21m11.316015302s) [noise=true , address=145.79.227.7:4295 , piece_id=FFBRXB3H5IF5PIZOMDP72HCJ2DQ6GXBXY32WSX5VILVUIGBRAMEA, seg_pos={0 0}, upload=536634]: Uploading piece...
Dec 31 03:50:11.445454181 (612h21m11.316537226s) [noise=true , address=145.79.227.4:4187 , piece_id=TBNWZR6HUWF2EUHGBWNNSKLIFXHQJZDWZZ4HQUMZ7NMUATSK632A, seg_pos={0 0}, upload=536634]: Uploading piece...
Dec 31 03:50:11.446652342 (612h21m11.317735387s) [noise=true , address=45.91.116.5:4298 , piece_id=LENGJXX4IA756IYMCIGNOUSIR4NAVM2J4NGOKTJSBPTUBVGRXV2Q, seg_pos={0 0}, upload=536634]: Uploading piece...
Dec 31 03:50:11.447216085 (612h21m11.318299131s) [noise=true , address=145.79.227.9:4365 , piece_id=OXPPG6623PEQZ62P5PFRO4D42Q42ORCHXKELZH4CBRX34HN7GCOA, seg_pos={0 0}, upload=536634]: Uploading piece...
Dec 31 03:50:11.447267033 (612h21m11.318350078s) [noise=true , address=145.79.227.2:4115 , piece_id=HNODHFPV6KXLLNKTZFMXJKXQ3NRQGRJSRX3NK5JKI3LCYZKV6WSA, seg_pos={0 0}, upload=536634]: Uploading piece...
Dec 31 03:50:11.447801821 (612h21m11.318884866s) [noise=true , address=45.91.116.6:4975 , piece_id=3NZ65H2KH7UM6YVAJ3SGAQVM6N3265HRTY2SOVS5TKY2QGANTYDQ, seg_pos={0 0}, upload=536634]: Uploading piece...
Dec 31 03:50:11.447919815 (612h21m11.31900286s) [noise=true , address=145.79.227.34:4905 , piece_id=ED3GS7G4OGNYCS2ZMHSY2IATCSJBOBZPY7NHAITMQ7NDARWAHKBA, seg_pos={0 0}, upload=536634]: Uploading piece...
Dec 31 03:50:11.448229615 (612h21m11.31931266s) [noise=true , address=145.79.227.18:4007 , piece_id=2EEZQUUWUHNLLL66MFCNRYGFLN5DGRULX5DE6PZBOCV6ZFLBUJXA, seg_pos={0 0}, upload=536634]: Uploading piece...
Dec 31 03:50:11.448703548 (612h21m11.319786593s) [noise=true , address=77.90.31.12:4590 , piece_id=HOUB4OTVNXMR24WHVR333HLX7I42YWADGNCZDNHHYMZW5ZA2OFQQ, seg_pos={0 0}, upload=536634]: Uploading piece...
Dec 31 03:50:11.448873221 (612h21m11.319956266s) [noise=true , address=145.79.227.25:4617 , piece_id=QFDBMVWHFJ6EVDCRHIUD45B7JK6UBWEOP7LAX3NCAUQXMUAO57KQ, seg_pos={0 0}, upload=536634]: Uploading piece...
Dec 31 03:50:11.449483112 (612h21m11.320566157s) [noise=true , address=77.90.31.19:4842 , piece_id=B2VDLOZNXW2KH3YDN55SCB2S5CDUXG7IZ4OVO3WSHECEDRVUMRLQ, seg_pos={0 0}, upload=536634]: Uploading piece...
Dec 31 03:50:11.451671970 (612h21m11.322755015s) [noise=true , address=77.90.31.10:4518 , piece_id=ZF2F423K6MLCHS5NQOGD4RAF7ENMUSBXREHTWWHS757AE454M3VQ, seg_pos={0 0}, upload=536634]: Uploading piece...
Dec 31 03:50:11.452259479 (612h21m11.323342524s) [noise=true , address=77.90.31.13:4626 , piece_id=6QYHJINPA3RLVO4XDNJ246GKO6BUX4M3LVWAMYK7QDCOMHS57E4A, seg_pos={0 0}, upload=536634]: Uploading piece...
Dec 31 03:50:11.452895740 (612h21m11.323978785s) [noise=true , address=145.79.227.10:4402 , piece_id=V3JOL33VJTN7U6F2OTSYBDOBVDHHP2FG45YP2XRSFYXLGHDOAPLQ, seg_pos={0 0}, upload=536634]: Uploading piece...
Dec 31 03:50:11.452955685 (612h21m11.32403872s) [noise=true , address=145.79.227.39:4013 , piece_id=LWU44VYKNIP7SJC4OPPE5R2BQOAPV7JLF3VPKYBCP66RJK7BYIDA, seg_pos={0 0}, upload=536634]: Uploading piece...
Dec 31 03:50:13.053903825 (612h21m12.924986871s) [upload=536634]: Next returned nil segment
Dec 31 03:50:13.053921288 (612h21m12.925004353s) [upload=536634]: Waiting for error group managing segments...
Dec 31 03:50:13.435484673 (612h21m13.306567718s) [noise=true , address=145.79.227.2:4115 , piece_id=HNODHFPV6KXLLNKTZFMXJKXQ3NRQGRJSRX3NK5JKI3LCYZKV6WSA, seg_pos={0 0}, upload=536634]: Done uploading piece. err: <nil>
Dec 31 03:50:13.438177049 (612h21m13.309260084s) [noise=true , address=145.79.227.18:4007 , piece_id=2EEZQUUWUHNLLL66MFCNRYGFLN5DGRULX5DE6PZBOCV6ZFLBUJXA, seg_pos={0 0}, upload=536634]: Done uploading piece. err: <nil>
Dec 31 03:50:13.448513851 (612h21m13.319596886s) [noise=true , address=145.79.227.7:4295 , piece_id=FFBRXB3H5IF5PIZOMDP72HCJ2DQ6GXBXY32WSX5VILVUIGBRAMEA, seg_pos={0 0}, upload=536634]: Done uploading piece. err: <nil>
Dec 31 03:50:13.452282387 (612h21m13.323365422s) [noise=true , address=145.79.227.4:4187 , piece_id=TBNWZR6HUWF2EUHGBWNNSKLIFXHQJZDWZZ4HQUMZ7NMUATSK632A, seg_pos={0 0}, upload=536634]: Done uploading piece. err: <nil>
Dec 31 03:50:13.454449504 (612h21m13.325532549s) [noise=true , address=145.79.227.25:4617 , piece_id=QFDBMVWHFJ6EVDCRHIUD45B7JK6UBWEOP7LAX3NCAUQXMUAO57KQ, seg_pos={0 0}, upload=536634]: Done uploading piece. err: <nil>
Dec 31 03:50:13.455594474 (612h21m13.326677519s) [noise=true , address=145.79.227.39:4013 , piece_id=LWU44VYKNIP7SJC4OPPE5R2BQOAPV7JLF3VPKYBCP66RJK7BYIDA, seg_pos={0 0}, upload=536634]: Done uploading piece. err: <nil>
Dec 31 03:50:13.457975968 (612h21m13.329059013s) [noise=true , address=145.79.227.10:4402 , piece_id=V3JOL33VJTN7U6F2OTSYBDOBVDHHP2FG45YP2XRSFYXLGHDOAPLQ, seg_pos={0 0}, upload=536634]: Done uploading piece. err: <nil>
Dec 31 03:50:13.521418281 (612h21m13.392501316s) [noise=true , address=145.79.227.9:4365 , piece_id=OXPPG6623PEQZ62P5PFRO4D42Q42ORCHXKELZH4CBRX34HN7GCOA, seg_pos={0 0}, upload=536634]: Done uploading piece. err: <nil>
Dec 31 03:50:13.712710334 (612h21m13.583793379s) [noise=true , address=77.90.31.13:4626 , piece_id=6QYHJINPA3RLVO4XDNJ246GKO6BUX4M3LVWAMYK7QDCOMHS57E4A, seg_pos={0 0}, upload=536634]: Done uploading piece. err: <nil>
Dec 31 03:50:13.765619068 (612h21m13.636702113s) [noise=true , address=45.91.116.5:4298 , piece_id=LENGJXX4IA756IYMCIGNOUSIR4NAVM2J4NGOKTJSBPTUBVGRXV2Q, seg_pos={0 0}, upload=536634]: Done uploading piece. err: <nil>
Dec 31 03:50:13.799377839 (612h21m13.670460874s) [noise=true , address=77.90.31.10:4518 , piece_id=ZF2F423K6MLCHS5NQOGD4RAF7ENMUSBXREHTWWHS757AE454M3VQ, seg_pos={0 0}, upload=536634]: Done uploading piece. err: <nil>
Dec 31 03:50:13.804835250 (612h21m13.675918295s) [noise=true , address=45.91.116.8:4045 , piece_id=CVUQDF5JIRWOUW57SLUOKGL4EFKN4JVSLOK5YLK6V3DQD3VZXXUQ, seg_pos={0 0}, upload=536634]: Done uploading piece. err: <nil>
Dec 31 03:50:13.841658255 (612h21m13.7127413s) [noise=true , address=77.90.31.12:4590 , piece_id=HOUB4OTVNXMR24WHVR333HLX7I42YWADGNCZDNHHYMZW5ZA2OFQQ, seg_pos={0 0}, upload=536634]: Done uploading piece. err: <nil>
Dec 31 03:50:13.841676751 (612h21m13.712759786s) [seg_pos={0 0}, upload=536634]: Segment reached optimal threshold of 13 pieces.
Dec 31 03:50:13.841886240 (612h21m13.712969275s) [noise=true , address=145.79.227.34:4905 , piece_id=ED3GS7G4OGNYCS2ZMHSY2IATCSJBOBZPY7NHAITMQ7NDARWAHKBA, seg_pos={0 0}, upload=536634]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9): protocol: expected piece hash; context canceled; EOF
Dec 31 03:50:13.841893133 (612h21m13.712976168s) [noise=true , address=145.79.227.34:4905 , piece_id=ED3GS7G4OGNYCS2ZMHSY2IATCSJBOBZPY7NHAITMQ7NDARWAHKBA, seg_pos={0 0}, upload=536634]: Long tail cancellation due to optimal threshold
Dec 31 03:50:13.841978385 (612h21m13.71306142s) [noise=true , address=45.91.116.6:4975 , piece_id=3NZ65H2KH7UM6YVAJ3SGAQVM6N3265HRTY2SOVS5TKY2QGANTYDQ, seg_pos={0 0}, upload=536634]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF
Dec 31 03:50:13.841986200 (612h21m13.713069245s) [noise=true , address=45.91.116.6:4975 , piece_id=3NZ65H2KH7UM6YVAJ3SGAQVM6N3265HRTY2SOVS5TKY2QGANTYDQ, seg_pos={0 0}, upload=536634]: Long tail cancellation due to optimal threshold
Dec 31 03:50:13.842138029 (612h21m13.713221074s) [noise=true , address=77.90.31.19:4842 , piece_id=B2VDLOZNXW2KH3YDN55SCB2S5CDUXG7IZ4OVO3WSHECEDRVUMRLQ, seg_pos={0 0}, upload=536634]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF
Dec 31 03:50:13.842147297 (612h21m13.713230332s) [noise=true , address=77.90.31.19:4842 , piece_id=B2VDLOZNXW2KH3YDN55SCB2S5CDUXG7IZ4OVO3WSHECEDRVUMRLQ, seg_pos={0 0}, upload=536634]: Long tail cancellation due to optimal threshold
Dec 31 03:50:13.842205848 (612h21m13.713288893s) [seg_pos={0 0}, upload=536634]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 31 03:50:13.842322991 (612h21m13.713406026s) [upload=536634]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 31 03:50:13.842329124 (612h21m13.713412159s) [upload=536634]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 31 03:50:14.152875101 (612h21m14.023958146s) [upload=536634]: Done uploading segments...
Dec 31 03:50:14.152919676 (612h21m14.024002721s) [upload=536634]: Upload finished. err: <nil>
Duration: 30.003484087s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
13 out of 60 is failed (threshold=6) [.......X.X...X...X...X.X.X...X......X..XX.....X....X........] duration:
3.745551191s trace:
https://jaeger.dev.storj.io/trace/255fcfcc92407481
FAIL (2025-12-31 03:50:22 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.74567188s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [..........................X.................................] duration:
53.44987511s trace:
https://jaeger.dev.storj.io/trace/634c422994952cfc
FAIL (2025-12-31 03:50:17 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 121Xa3rvgKQhHc5NBufs57Ybnw3fYKUjgRyK32TTwWoi4E8m57o, Piece ID: 7ZJODSC7K4XWDIBJGMYHMGRFCXZGHTSGWTWPO7NKPWGGT73NFGOQ) context deadline exceeded; piecestore: (Node ID: 12caEiJobigXcGWwaX3Vt16zd2NXqk1WzLWxZME44E8Q8wdhn3q, Piece ID: RTZZG5ZDBPM74ZXITWZZ6LKJCKUOQ3YY4UUTMZ5VRVKFZE4YKAUA) context deadline exceeded; piecestore: (Node ID: 1zwuzCLWy2q166Ld52WUG9fvXNdJWsQrg2NearHYg7pyxpXFen, Piece ID: EJWBT3NCXTW5JSOIICAASNNDWFMIM2AO2SAE26S5POMMLEOYUH7Q) context deadline exceeded
======LOGS======
) [noise=true , address=193.149.164.143:4704 , piece_id=XIHZWOG63PEC3AQOHHWBLF53Y3B42NOO6TRNEUR3CIJB3YZTX3LQ, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.363237580 (611h51m51.234320615s) [noise=true , address=145.79.227.15:4889 , piece_id=EACS3IK7KTZI73UJEUOZB33TZN64WDWRC7TV5YJZ5ZMQVATQ3ORQ, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.364121825 (611h51m51.23520486s) [noise=true , address=193.149.164.143:4699 , piece_id=5LCUJBU4MEYEA7CQCA2FVVX6VCEX7SU3W4N3BBPQMLP6AHG5ZKKQ, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.364545251 (611h51m51.235628286s) [noise=true , address=45.91.116.11:4143 , piece_id=RTZZG5ZDBPM74ZXITWZZ6LKJCKUOQ3YY4UUTMZ5VRVKFZE4YKAUA, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.364959129 (611h51m51.236042174s) [noise=true , address=145.79.227.3:4176 , piece_id=TBPK43ULJ75RKDV5P63GXNVYZWG7TIQ3YLKMBLU55OQ2PFGSBH7A, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.365381323 (611h51m51.236464368s) [noise=true , address=145.79.227.44:4217 , piece_id=B5JSD2DUWSMSEME5CZCWFBXELK4WRMAXLLA2QQL73SB6WR6ECXUA, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.367027327 (611h51m51.238110362s) [noise=true , address=77.90.31.11:4561 , piece_id=IRE2JNK63DRH52RXBC5D3MRL6QO5S5N44UFVFWDRNY7LZBQ3IFMQ, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.367801231 (611h51m51.238884326s) [noise=true , address=145.79.227.40:4058 , piece_id=ECUZDPLHYQW3KYSTHKWBPD3CCTUCMAZ6XMOSH5S46ACF2T5366BQ, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.369086348 (611h51m51.240169404s) [noise=true , address=77.90.31.10:4536 , piece_id=EJWBT3NCXTW5JSOIICAASNNDWFMIM2AO2SAE26S5POMMLEOYUH7Q, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.370451017 (611h51m51.241534062s) [noise=true , address=77.90.31.19:4861 , piece_id=U7I2SDPMKDL4RLP2CIWZUSCEWPM5LP67VMUOLFKBHHVSCXL55MYQ, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.371980618 (611h51m51.243063654s) [noise=true , address=64.49.13.26:4955 , piece_id=53IGVDHKGGBIEKVJ6A52Q647E74KVI4FFRQU7VYK7POFU4CCX4HA, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.374009812 (611h51m51.245092847s) [noise=true , address=77.90.31.14:4671 , piece_id=2YXJ5RJERCNTD3TJHJHF4WBHR3DEOHDIOWI53H4PDS4EWQECEU2A, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.374500948 (611h51m51.245583993s) [noise=true , address=77.90.31.13:4632 , piece_id=72ADTLC2JC7Z3Z4GYKPECNAK6QMCMONGLUKZKQFLWN5UIQ4IPZYQ, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.392662589 (611h51m51.263745634s) [noise=true , address=145.79.227.32:4865 , piece_id=ZXAGYGEC35VVET2IAX2KJII4PUK3DFQXOFAYCC5R7KV5PNQCSUIA, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.393651513 (611h51m51.264734558s) [noise=true , address=145.79.227.31:4804 , piece_id=4E3PGEBPB2BFUAHNXJI4MZFWNNNZMMFDF5UBTT2UM4F2NHOHCIQQ, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:51.396266312 (611h51m51.267349357s) [noise=true , address=145.79.227.29:4966 , piece_id=N4JGQH23KY2BMVZZWWX7CKZN2FC2FJBFNAZEGZ7CRMRQAYEAZJLA, seg_pos={0 0}, upload=536207]: Uploading piece...
Dec 31 03:20:52.828565815 (611h51m52.69964886s) [upload=536207]: Next returned nil segment
Dec 31 03:20:52.828580594 (611h51m52.699663629s) [upload=536207]: Waiting for error group managing segments...
Dec 31 03:20:53.204430167 (611h51m53.075513202s) [noise=true , address=145.79.227.15:4889 , piece_id=EACS3IK7KTZI73UJEUOZB33TZN64WDWRC7TV5YJZ5ZMQVATQ3ORQ, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:53.265960889 (611h51m53.137043924s) [noise=true , address=145.79.227.31:4804 , piece_id=4E3PGEBPB2BFUAHNXJI4MZFWNNNZMMFDF5UBTT2UM4F2NHOHCIQQ, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:53.266193351 (611h51m53.137276386s) [noise=true , address=145.79.227.32:4865 , piece_id=ZXAGYGEC35VVET2IAX2KJII4PUK3DFQXOFAYCC5R7KV5PNQCSUIA, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:53.331763405 (611h51m53.20284644s) [noise=true , address=5.231.73.2:4035 , piece_id=7ZJODSC7K4XWDIBJGMYHMGRFCXZGHTSGWTWPO7NKPWGGT73NFGOQ, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:53.412602422 (611h51m53.283685467s) [noise=true , address=77.90.31.10:4536 , piece_id=EJWBT3NCXTW5JSOIICAASNNDWFMIM2AO2SAE26S5POMMLEOYUH7Q, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:53.419806240 (611h51m53.290889275s) [noise=true , address=77.90.31.14:4671 , piece_id=2YXJ5RJERCNTD3TJHJHF4WBHR3DEOHDIOWI53H4PDS4EWQECEU2A, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:53.498590626 (611h51m53.369673671s) [noise=true , address=77.90.31.11:4561 , piece_id=IRE2JNK63DRH52RXBC5D3MRL6QO5S5N44UFVFWDRNY7LZBQ3IFMQ, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:53.520983294 (611h51m53.392066339s) [noise=true , address=45.91.116.11:4143 , piece_id=RTZZG5ZDBPM74ZXITWZZ6LKJCKUOQ3YY4UUTMZ5VRVKFZE4YKAUA, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:53.585203506 (611h51m53.456286541s) [noise=true , address=77.90.31.19:4861 , piece_id=U7I2SDPMKDL4RLP2CIWZUSCEWPM5LP67VMUOLFKBHHVSCXL55MYQ, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:54.188950073 (611h51m54.060033119s) [noise=true , address=145.79.227.44:4217 , piece_id=B5JSD2DUWSMSEME5CZCWFBXELK4WRMAXLLA2QQL73SB6WR6ECXUA, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:54.204771008 (611h51m54.075854053s) [noise=true , address=145.79.227.40:4058 , piece_id=ECUZDPLHYQW3KYSTHKWBPD3CCTUCMAZ6XMOSH5S46ACF2T5366BQ, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:54.231988834 (611h51m54.103071879s) [noise=true , address=193.149.164.143:4699 , piece_id=5LCUJBU4MEYEA7CQCA2FVVX6VCEX7SU3W4N3BBPQMLP6AHG5ZKKQ, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:54.275280076 (611h51m54.146363121s) [noise=true , address=145.79.227.29:4966 , piece_id=N4JGQH23KY2BMVZZWWX7CKZN2FC2FJBFNAZEGZ7CRMRQAYEAZJLA, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:54.307143628 (611h51m54.178226673s) [noise=true , address=77.90.31.12:4602 , piece_id=ZPVIMHS5CQAKREW456SPNTHR3WM6IWZDB6AJZIYFREALJ6V63O6A, seg_pos={0 0}, upload=536207]: Done uploading piece. err: <nil>
Dec 31 03:20:54.307168195 (611h51m54.17825124s) [seg_pos={0 0}, upload=536207]: Segment reached optimal threshold of 14 pieces.
Dec 31 03:20:54.307445734 (611h51m54.178528789s) [noise=true , address=77.90.31.13:4632 , piece_id=72ADTLC2JC7Z3Z4GYKPECNAK6QMCMONGLUKZKQFLWN5UIQ4IPZYQ, seg_pos={0 0}, upload=536207]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13iPed9zYrthvh36xo3k4gsqm4tz86iVkNvAr3wauuArQLquyS): protocol: expected piece hash; context canceled; EOF
Dec 31 03:20:54.307457145 (611h51m54.1785402s) [noise=true , address=77.90.31.13:4632 , piece_id=72ADTLC2JC7Z3Z4GYKPECNAK6QMCMONGLUKZKQFLWN5UIQ4IPZYQ, seg_pos={0 0}, upload=536207]: Long tail cancellation due to optimal threshold
Dec 31 03:20:54.307494436 (611h51m54.178577471s) [noise=true , address=193.149.164.143:4704 , piece_id=XIHZWOG63PEC3AQOHHWBLF53Y3B42NOO6TRNEUR3CIJB3YZTX3LQ, seg_pos={0 0}, upload=536207]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bUVmWsfAfQiPWhDkYRNGhims8TLQqGqvhrZMYugLLTdmLk2xY): protocol: expected piece hash; context canceled; EOF
Dec 31 03:20:54.307513032 (611h51m54.178596067s) [noise=true , address=193.149.164.143:4704 , piece_id=XIHZWOG63PEC3AQOHHWBLF53Y3B42NOO6TRNEUR3CIJB3YZTX3LQ, seg_pos={0 0}, upload=536207]: Long tail cancellation due to optimal threshold
Dec 31 03:20:54.307792985 (611h51m54.17887602s) [noise=true , address=145.79.227.3:4176 , piece_id=TBPK43ULJ75RKDV5P63GXNVYZWG7TIQ3YLKMBLU55OQ2PFGSBH7A, seg_pos={0 0}, upload=536207]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kGgrsNn5tv2GkcuVHHArCACwRB2qwbj3X3JmzVF4UL6dLRW1m): protocol: write tcp 188.166.225.170:39612->145.79.227.3:4176: use of closed network connection; write tcp 188.166.225.170:39612->145.79.227.3:4176: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39612->145.79.227.3:4176: use of closed network connection
Dec 31 03:20:54.307803214 (611h51m54.178886259s) [noise=true , address=145.79.227.3:4176 , piece_id=TBPK43ULJ75RKDV5P63GXNVYZWG7TIQ3YLKMBLU55OQ2PFGSBH7A, seg_pos={0 0}, upload=536207]: Long tail cancellation due to optimal threshold
Dec 31 03:20:54.307870673 (611h51m54.178953718s) [noise=true , address=64.49.13.26:4955 , piece_id=53IGVDHKGGBIEKVJ6A52Q647E74KVI4FFRQU7VYK7POFU4CCX4HA, seg_pos={0 0}, upload=536207]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125TV3v1CzV1Q9TPfyVHsn26NZ5XcHixKxA66t96iz62FF4SHAP): protocol: expected piece hash; context canceled; EOF
Dec 31 03:20:54.307882294 (611h51m54.178965339s) [noise=true , address=64.49.13.26:4955 , piece_id=53IGVDHKGGBIEKVJ6A52Q647E74KVI4FFRQU7VYK7POFU4CCX4HA, seg_pos={0 0}, upload=536207]: Long tail cancellation due to optimal threshold
Dec 31 03:20:54.307909116 (611h51m54.178992151s) [seg_pos={0 0}, upload=536207]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 31 03:20:54.308005189 (611h51m54.179088234s) [upload=536207]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 31 03:20:54.308013014 (611h51m54.179096049s) [upload=536207]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 31 03:20:54.593642638 (611h51m54.464725683s) [upload=536207]: Done uploading segments...
Dec 31 03:20:54.593663947 (611h51m54.464746983s) [upload=536207]: Upload finished. err: <nil>
Duration: 53.449891171s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [........................X....X.X..X.........................] duration:
7.215877186s trace:
https://jaeger.dev.storj.io/trace/5e7c34cde28aa313
FAIL (2025-12-31 03:50:01 UTC)
orbiter warning: context deadline exceeded
Duration: 7.21591058s
edge
gateway
us-select-1
|